Using internal S3 provider ... and something funky is going on!

[a2geek]

I'd like to use Fog to connect to an internal S3 provider (custom host).

What I've been finding is that some operations work but others mysteriously connect to s3.amazonaws.com.

Pulling some samples together, I can write and read from a bucket but not list buckets - that mysteriously switches over to the AWS url.

Digging a little further, I see that get_service.rb (see here) seems to be the only place where the Amazon AWS URL is assigned except outside of the common setup logic. I'd submit a PR but I also see the last comment was "set correct host for get service operation" -- so I wonder if actual AWS S3 connections require something different than non-Amazon connections.

My sample code, in case that is relevant:

require 'fog/aws'

print "connecting...\n"
connection = Fog::AWS::Storage.new({
    :aws_access_key_id =>     'ACCESS_KEY',
    :aws_secret_access_key => 'SECRET_KEY',
    :region =>                'us-east-1',
    :endpoint =>              'https://s3.internal.host:443',
    :path_style =>            true,
    :connection_options =>    { ssl_verify_peer: false },
})

# put_object operation
print "put object...\n"
connection.put_object(
        'cf-buildpack',
        'testobject',
        "Hello world!! #{Time.now}",
        content_type: 'text/plain'
)

# get_object operation
print "get object...\n"
download_testobject = connection.get_object(
         'cf-buildpack',
         'testobject'
).body
print "Downloaded 'testobject' as  '#{download_testobject}'.\n"

# list directories
print "list buckets...\n"
p connection.directories

The questionable code in Fog::AWS::Storage::Real#get_service

        def get_service
          request({
            :expects  => 200,
            :headers  => {},
            :host     => 's3.amazonaws.com',    #  <== this seems to be the issue
            :idempotent => true,
            :method   => 'GET',
            :parser   => Fog::Parsers::AWS::Storage::GetService.new
          })
        end

When I comment out host, it fixes my issue.

The stack trace I get is:

list buckets...
Traceback (most recent call last):
    39: from example.rb:35:in `<main>'
    38: from example.rb:35:in `p'
    37: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:72:in `inspect'
    36: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:14:in `format'
    35: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:41:in `indent'
    34: from /var/lib/gems/2.5.0/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
    33: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:14:in `block in format'
    32: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:54:in `object_string'
    31: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:72:in `nested_objects_string'
    30: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:41:in `indent'
    29: from /var/lib/gems/2.5.0/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
    28: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:72:in `block in nested_objects_string'
    27: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/formatador.rb:85:in `inspect_object'
    26: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:16:in `map'
    25: from /var/lib/gems/2.5.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:112:in `lazy_load'
    24: from /var/lib/gems/2.5.0/gems/fog-aws-3.6.7/lib/fog/aws/models/storage/directories.rb:10:in `all'
    23: from /var/lib/gems/2.5.0/gems/fog-aws-3.6.7/lib/fog/aws/requests/storage/get_service.rb:21:in `get_service'
    22: from /var/lib/gems/2.5.0/gems/fog-aws-3.6.7/lib/fog/aws/storage.rb:623:in `request'
    21: from /var/lib/gems/2.5.0/gems/fog-aws-3.6.7/lib/fog/aws/storage.rb:628:in `_request'
    20: from /var/lib/gems/2.5.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
    19: from /var/lib/gems/2.5.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
    18: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:232:in `request'
    17: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:306:in `rescue in request'
    16: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/base.rb:17:in `error_call'
    15: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/base.rb:17:in `error_call'
    14: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/idempotent.rb:50:in `error_call'
    13: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:232:in `request'
    12: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:306:in `rescue in request'
    11: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/base.rb:17:in `error_call'
    10: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/base.rb:17:in `error_call'
     9: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/idempotent.rb:50:in `error_call'
     8: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:232:in `request'
     7: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:306:in `rescue in request'
     6: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/base.rb:17:in `error_call'
     5: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/base.rb:17:in `error_call'
     4: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/idempotent.rb:50:in `error_call'
     3: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:281:in `request'
     2: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/connection.rb:450:in `response'
     1: from /var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/response_parser.rb:12:in `response_call'
/var/lib/gems/2.5.0/gems/excon-0.76.0/lib/excon/middlewares/expects.rb:13:in `response_call': Expected(200) <=> Actual(403 Forbidden) (Excon::Error::Forbidden)
excon.error.response
  :body          => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>cf-access-key</AWSAccessKeyId><RequestId>70B363F2EA5C4431</RequestId><HostId>sx6fl/LarJY5QnWFrMFgQxdpi0L03yHfekmVFXBRA47+zl2XhSfgDn9NfK5mxiaf2mdCtnU8vv8=</HostId></Error>"
  :cookies       => [
  ]
  :headers       => {
    "Content-Type"     => "application/xml"
    "Date"             => "Fri, 04 Sep 2020 02:41:08 GMT"
    "Server"           => "AmazonS3"
    "x-amz-id-2"       => "sx6fl/LarJY5QnWFrMFgQxdpi0L03yHfekmVFXBRA47+zl2XhSfgDn9NfK5mxiaf2mdCtnU8vv8="
    "x-amz-request-id" => "70B363F2EA5C4431"
  }
  :host          => "s3.amazonaws.com"
  :local_address => "192.168.5.254"
  :local_port    => 49984
  :path          => "/"
  :port          => 443
  :reason_phrase => "Forbidden"
  :remote_ip     => "52.216.20.43"
  :status        => 403
  :status_line   => "HTTP/1.1 403 Forbidden\r\n"

For clarity, nowhere do I specify s3.amazonaws.com in the sample code.

Thanks!

[geemus]

@a2geek - Thanks for the detailed report, it definitely looks like that is causing the problem. If I recall it has to do with s3 wanting that specific host (rather than one of the regionalized hosts). If a non-aws url is specified it could use whatever the host is (as it did prior to that patch), but I think for AWS it should probably maintain the same behavior. Does that make sense?

[a2geek]

Agreed. Uncertain how to accomplish that. I'm pretty certain I saw that setting endpoint internally sets host, port, and protocol (going by memory so reality may vary a bit). So I think @host is always set? Maybe figure out if it's an S3 endpoint and act accordingly?

[geemus]

Yeah, I think checking host probably would make sense. And then we could have s3 behavior only if it is either *.amazonaws.com or *.amazonaws.com.cn. Does that line up with what you were thinking?

[a2geek]

Yeah, sounds like a decent strategy.

[github-actions[bot]]

This issue has been marked inactive and will be closed if no further activity occurs.

[max-critcrew]

Hey,

has this ever been resolved? We're currently having exactly this issue using the fog-aws version together with a self-hosted GitLab instance. We're trying to use B2/Backblaze S3 compatible API. Also see here: https://docs.gitlab.com/ee/administration/object_storage.html

[a2geek]

@max-critcrew - Not that I'm aware of. I don't recall how I got around it but I ultimately ended up using a different tool to setup connections. (Note that I'm not a Ruby programmer and I was configuring Cloud Foundry, so that solution is likely not useful to you.)

Rereading the discussion, I think I had an idea where the issue occurred... and it sounds like the Fog team was willing to look at a PR...

[a2geek]

Oh, one additional note. I poked around in the Genesis configs for MinIO, and I see how they setup the region... maybe that is the trick? (I've moved on so can't really try this myself any more.)

  fog_connection:
    provider: AWS
    endpoint: (( grab params.blobstore_minio_endpoint ))
    aws_access_key_id: ((blobstore_access_key_id))
    aws_secret_access_key: ((blobstore_secret_access_key))
    aws_signature_version: '2'
    region: "''"     # <-- THIS??
    path_style: true

From: https://github.com/genesis-community/cf-genesis-kit/blob/a3cc0ab6c0896180d36462ecf8ae010b211c8122/overlay/blobstore/minio.yml#L11

[geemus]

@a2geek thanks for the extra context. I could imagine the region trick may have been needed, but I think with other more recent changes that setting the endpoint should be good-enough. I also discussed more with @max-critcrew on another issue and it sounds like he at least found a workaround for his particular case.

[a2geek]

Wonderful. Thanks for the update!