Custom S3 endpoints not working?

[jonathanxie]

Environment

• Elixir & Erlang versions (elixir --version):

Erlang/OTP 21 [erts-10.2.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.8.1 (compiled with Erlang/OTP 21)

• ex_aws 2.1.0 (Hex package) (mix) locked at 2.1.0 (ex_aws) b9265152
• ex_aws_s3 2.0.1 (Hex package) (mix) locked at 2.0.1 (ex_aws_s3) 9e09366e
• hackney 1.15.1 (Hex package) (rebar3) locked at 1.15.1 (hackney) 9f8f471c

Current behavior

I'm using Wasabi (S3 compliant) instead of S3 and I need to have a custom endpoint.

Setting ex_aws config doesn't seem to take effect and I get default values host and auth still.

Here's my config:

config :ex_aws, scheme: "https://", host: "s3.wasabisys.com", region: "us-east-1", access_key_id: ["MY_ACCESS_KEY_ID", :instance_role], secret_access_key: ["MY_SECRET_ACCESS_KEY", :instance_role], debug_requests: true

Here's the output from debugging requests:

[debug] Request HEADERS: [

{"Authorization", "AWS4-HMAC-SHA256 Credential=MY_ACCESS_KEY_ID/20190412/us-east-1/s3/aws4_request,SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date,Signature=asdf13123asefasdf1231asdfasasdfa1231asdfasf"}, 
{"host", "s3.amazonaws.com"}, 

{"x-amz-date", "20190412T005934Z"}, 

{"content-length", 70}, 
{"x-amz-content-sha256", "asdfasdff92a8b80aab02d9bf1a12e49ffcae424e4013a1c4f68b67e3d2asdfg"}

]

Expected behavior

Accessing wasabi has no issues if I use AWS SDKs. For node.js, I did this:

var AWS = require('aws-sdk');

var accessKeyId = 'MY_ACCESS_KEY_ID'; var secretAccessKey = 'MY_SECRET_ACCESS_KEY';

var wasabiEndpoint = new AWS.Endpoint('s3.us-west-1.wasabisys.com');

var s3 = new AWS.S3({ endpoint: wasabiEndpoint, accessKeyId: accessKeyId, secretAccessKey: secretAccessKey });

The host should be: s3.us-west-1.wasabisys.com

[bcat-eu]

Facing the same issue, if I change the priv/endpoints.exs and add a new endpoint (that passes "^(us|eu|ap|sa|ca)\-\w+\-\d+$" regex check) with custom hostname - it works, but would require forking the library.

One way around it that I found is using runtime overrides, something like this:

  defp overrides(opts \\ []) do
    Keyword.merge([host: Application.get_env(:ex_aws, :host)], opts)
  end

and then use it as the second parameter when making calls to ExAws.request/2 or ExAws.Config.new/2.

[philippneugebauer]

@benwilson512 wouldn't it make sense to be able to deactivate the region regex check or add a warning instead to support the various s3 providers that exist nowadays? Or any other better option that allows to use them?

[philippneugebauer]

Okay, it actually works for me in combination of Arc and ExAws with Wasabi. So this can be closed.

That's my ex_aws config part:

config :ex_aws,
  region: "eu-central-1",
  s3: [
    scheme: "https://",
    host: "s3.eu-central-1.wasabisys.com",
    region: "eu-central-1"
  ]

[philippneugebauer]

I still would argue that this config part should be documented somewhere. Because I did only find out about it in the Arc documentation.

[ijunaid8989]

I have a local endpoint without ssl.

  host: "localhost:8333"

But it gives me such error

2:50:44.863 [warn]  ExAws: HTTP ERROR: :nxdomain for URL: "https://[localhost:8333]/" ATTEMPT: 1

how can i drop http(s)

[benwilson512]

scheme: "http://"

[ijunaid8989]

with this, it goes like this

23:00:03.537 [warn]  ExAws: HTTP ERROR: :nxdomain for URL:
"http://[localhost:8333]:443/"
ATTEMPT: 1

Junaid Farooq Software Developer - Evercam.io

On Thu, 4 Jun 2020 at 22:59, Ben Wilson notifications@github.com wrote:

scheme: "http://"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ex-aws/ex_aws/issues/628#issuecomment-639012570, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHPBD77PVCQLPLQVRWEZZDRU7OGPANCNFSM4HFMYMOQ .

[benwilson512]

Please use the elixir forums or other elixir help areas for assistance, not the issue tracker, thanks

[redheli]

@ijunaid8989

 host: "localhost:",
 port: "8333"