Request for box's Amazon S3 region was denied

stevenscg commented 6 years ago

I have been using this plugin without issue for several years. But recently, we've started to run into the following error with Vagrant v2.1.5 and probably Vagrant v2.1.4. Plugin version 1.3.2.

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'my-box.box' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
Request for box's Amazon S3 region was denied.

This usually indicates that your user account is misconfigured. Ensure
your IAM policy allows the "s3:GetBucketLocation" action for your bucket:

    arn:aws:s3:::mybucket

The actual bucket is in us-west-2 and has always been.

We have tried multiple formats of the url per the recommendations in the README.

We are using .aws/credentials and .aws/config files per AWS recommendations.

We can download using the aws s3 cp .... syntax on the command line with same credentials.

I've looked at our IAM permissions several times. They appear to be correct and have not changed in several years of using this plugin.

The bucket is owned by another AWS account and IAM policies in both accounts appear to be correct.

I am still looking for the issue and will update this if I find anything.

stevenscg commented 6 years ago

If I try uploading this to a bucket also in us-west-2, but this time owned by the same AWS account of the user, everything works:

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'my-box.box' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Box file was not detected as metadata. Adding it directly...
==> default: Adding box 'my-box.box' (v0) for provider: virtualbox
    default: Downloading: https://s3-us-west-2.amazonaws.com/mybucket/my-box.box
    default: Signing S3 request with key 'XXXXXXXXXXXXXXXX' loaded from $AWS_ACCESS_KEY_ID

So, this leads me to think that the issue has to do with the cross-account IAM permissions and/or the bucket policy on the original bucket.

stevenscg commented 6 years ago

I am still fighting this issue and continue to suspect an issue accessing the bucket across accounts.

To rule out an issue with the age of the aws-sdk used with the plugin, I updated the gemspec to use version 2.11.150 of the aws-sdk.

I'm not familiar with the ruby toolchain, but it did seem to build and install. Behavior was unchanged.

I also changed the DEFAULT_REGION in util.rb to us-west-2 where my bucket is located. No change in behavior.

Further, I tried using the ENV['AWS_PROFILE'] credentials provider option as described in the project README, but saw no change in behavior.

stevenscg commented 6 years ago

Possible solution:

I noticed some issues with region handling in the past and in particular this change.

When I backed out the change and also while using version 2.11.150 of the aws-sdk-ruby, my download issues were resolved.

In my working configuration util.rb lines 32-34 are now:

      def self.s3_client(region = DEFAULT_REGION)
        ::Aws::S3::Client.new(region: region, force_path_style: true)
      end

@benesch any ideas why this works?

darrenob commented 5 years ago

Hello. Has there been any fix or workaround for this?

I'm also having great difficulty getting a Metadata box to download from s3 in a cross-account scenario due to "Request for box's Amazon S3 region was denied".

I'm using Vagrant 2.2.4, vagrant-s3auth 1.3.2

I'll give my issue here in case it helps.

The box is in an s3 bucket in Account A, region eu-west-1. My user (running Vagrant) is in Account B.

My user has access to the bucket through a combination of bucket policy and IAM roles as explained here by AWS

~$ cat ~/.aws/credentials
[default]
region = eu-west-1
aws_access_key_id = <KEY_ID>
aws_secret_access_key = <SECRET_KEY>

A bucket policy in Account A delegates access to Account B:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT-B:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::BUCKET"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT-B:root"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET",
                "arn:aws:s3:::BUCKET/*"
            ]
        }
    ]
}

My IAM user in Account B has a policy granting me access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::BUCKET"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET",
                "arn:aws:s3:::BUCKET/*"
            ]
        }
    ]
}

I can access the box manifest and the box it points to on the cli:

~$ aws s3 ls BUCKET/vagrant/BOX/BOX.json      
2019-04-18 18:59:50       7084 BOX.json      

~$ aws s3 ls BUCKET/vagrant/BOX/0.27.0/BOX.box
2019-04-18 18:59:50        364 BOX.box

I can also download it (GetObject):

~$ aws s3 cp s3://BUCKET/vagrant/BOX/0.27.0/BOX.box .
download: s3://BUCKET/vagrant/BOX/0.27.0/BOX.box to ./BOX.box

I can get its location

~$ aws s3api get-bucket-location --bucket BUCKET
eu-west-1

Yet the only way I can get Vagrant to access the box is to make both the manifest and the box it points to public.

Things I've tried:

In the Vagrantfile I've set

ENV["AWS_REGION"] = "eu-west-1"

I tried using different assumerole profile in the Vagrantfile

ENV["AWS_PROFILE"] = "account-a"

This profile is set up in my ~/.aws/config. All the aws cli commands above work using this profile.

I tried setting env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Same result.

I tried all the possible s3 bucket URL formats. Same result.

I should mention that the s3 objects are created and owned by a third account "Account C: where we build the images with Packer and upload using the vagrant-s3 plugin. We set "bucket-owner-full-contriol" ACL.

The object ACLs are as follows:

~$ aws s3api get-object-acl --bucket BUCKET --key vagrant/BOX/0.27.0/BOX.box
GRANTS  FULL_CONTROL
GRANTEE <Account C> <canonical-user-id> CanonicalUser
GRANTS  FULL_CONTROL
GRANTEE <Account A> <canonical-user-id> CanonicalUser
GRANTS  READ
GRANTEE <Account B> <canonical-user-id> CanonicalUser
GRANTS  READ_ACP
GRANTEE <Account B> <canonical-user-id> CanonicalUser
OWNER   <Account C> <canonical-user-id>

Here's the debug log output with redacted items capitalized.

==> vagrant-machine: Checking for updates to 'COMPANY/BOXNAME'
 INFO interface: detail: Latest installed version: 0.26.0
 INFO interface: detail:     vagrant-machine: Latest installed version: 0.26.0
    vagrant-machine: Latest installed version: 0.26.0
 INFO interface: detail: Version constraints:
 INFO interface: detail:     vagrant-machine: Version constraints:
    vagrant-machine: Version constraints:
 INFO interface: detail: Provider: aws
 INFO interface: detail:     vagrant-machine: Provider: aws
    vagrant-machine: Provider: aws
 INFO downloader: Downloader starting download:
 INFO downloader:   -- Source: https://BUCKET.s3-eu-west-1.amazonaws.com/vagrant/BOX/BOX.json
 INFO downloader:   -- Destination: /var/folders/wc/dks9b1dx1hb4ymnrx_rdxjdh0000gp/T/vagrant-load-metadata20190425-50786-1v4dhga
 INFO subprocess: Starting process: ["/opt/vagrant/embedded/bin/curl", "-q", "--fail", "--location", "--max-redirs", "10", "--verbose", "--user-agent", "Vagrant/2.2.4 (+https://www.vagrantup.com; ruby2.4.4) ", "-H", "Accept: application/json", "--output", "/var/folders/wc/dks9b1dx1hb4ymnrx_rdxjdh0000gp/T/vagrant-load-metadata20190425-50786-1v4dhga", "https://BUCKET.s3-eu-west-1.amazonaws.com/vagrant/BOX/BOX.json"]
 INFO subprocess: Command in the installer. Specifying DYLD_LIBRARY_PATH...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   T
DEBUG subprocess: stderr: otal   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG subprocess: stderr: *   Trying 52.218.16.67...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
DEBUG subprocess: stderr: * Connected to BUCKET.s3-eu-west-1.amazonaws.com (52.218.16.67) port 443 (#0)
DEBUG subprocess: stderr: * ALPN, offering http/1.1
DEBUG subprocess: stderr: * successfully set certificate verify locations:
*   CAfile: /opt/vagrant/embedded/cacert.pem
  CApath: none
DEBUG subprocess: stderr: } [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [248 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [87 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3139 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
DEBUG subprocess: stderr: * TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
DEBUG subprocess: stderr: * Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3-eu-west-1.amazonaws.com
*  start date: Nov  8 00:00:00 2018 GMT
*  expire date: Nov  6 12:00:00 2019 GMT
*  subjectAltName: host "BUCKET.s3-eu-west-1.amazonaws.com" matched cert's "*.s3-eu-west-1.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
} [5 bytes data]
> GET /vagrant/BOX/BOX.json HTTP/1.1
> Host: BUCKET.s3-eu-west-1.amazonaws.com
> User-Agent: Vagrant/2.2.4 (+https://www.vagrantup.com; ruby2.4.4)
> Accept: application/json
>
DEBUG subprocess: stderr: { [5 bytes data]
* The requested URL returned error: 403 Forbidden
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
* Closing connection 0
} [5 bytes data]
* TLSv1.2 (OUT), TLS alert, Client hello (1):
} [2 bytes data]
DEBUG subprocess: stderr: curl: (22) The requested URL returned error: 403 Forbidden
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 31997
DEBUG subprocess: Exit status: 22
 WARN downloader: Downloader exit code: 22
 INFO environment: Running hook: environment_unload
 INFO runner: Preparing hooks for middleware sequence...
 INFO runner: 2 hooks defined.
 INFO runner: Running action: environment_unload #<Vagrant::Action::Builder:0x00000001013c3cf8>
ERROR vagrant: Vagrant experienced an error! Details:
ERROR vagrant: #<VagrantPlugins::S3Auth::Errors::BucketLocationAccessDeniedError: Request for box's Amazon S3 region was denied.

This usually indicates that your user account is misconfigured. Ensure
your IAM policy allows the "s3:GetBucketLocation" action for your bucket:

    arn:aws:s3:::BUCKET>
ERROR vagrant: Request for box's Amazon S3 region was denied.

This usually indicates that your user account is misconfigured. Ensure
your IAM policy allows the "s3:GetBucketLocation" action for your bucket:

    arn:aws:s3:::BUCKET
ERROR vagrant: /Users/darren/.vagrant.d/gems/2.4.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/util.rb:72:in `rescue in get_bucket_region'
/Users/darren/.vagrant.d/gems/2.4.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/util.rb:67:in `get_bucket_region'
/Users/darren/.vagrant.d/gems/2.4.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/util.rb:54:in `s3_object_for'
/Users/darren/.vagrant.d/gems/2.4.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:35:in `s3auth_download'
/Users/darren/.vagrant.d/gems/2.4.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:77:in `rescue in execute_curl_with_s3auth'
/Users/darren/.vagrant.d/gems/2.4.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:71:in `execute_curl_with_s3auth'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/lib/vagrant/util/downloader.rb:111:in `download!'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/lib/vagrant/box.rb:135:in `load_metadata'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/lib/vagrant/box.rb:168:in `has_update?'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/plugins/commands/box/command/update.rb:141:in `box_update'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/plugins/commands/box/command/update.rb:131:in `block in update_vms'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/plugins/commands/box/command/update.rb:120:in `each'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/plugins/commands/box/command/update.rb:120:in `update_vms'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/plugins/commands/box/command/update.rb:49:in `execute'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/plugins/commands/box/command/root.rb:66:in `execute'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/lib/vagrant/cli.rb:66:in `execute'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/lib/vagrant/environment.rb:292:in `cli'
/opt/vagrant/embedded/gems/2.2.4/gems/vagrant-2.2.4/bin/vagrant:182:in `<main>'
 INFO interface: error: Request for box's Amazon S3 region was denied.

This usually indicates that your user account is misconfigured. Ensure
your IAM policy allows the "s3:GetBucketLocation" action for your bucket:

    arn:aws:s3:::BUCKET
Request for box's Amazon S3 region was denied.

This usually indicates that your user account is misconfigured. Ensure
your IAM policy allows the "s3:GetBucketLocation" action for your bucket:

    arn:aws:s3:::BUCKET
 INFO interface: Machine: error-exit ["VagrantPlugins::S3Auth::Errors::BucketLocationAccessDeniedError", "Request for box's Amazon S3 region was denied.\n\nThis usually indicates that your user account is misconfigured. Ensure\nyour IAM policy allows the \"s3:GetBucketLocation\" action for your bucket:\n\n    arn:aws:s3:::BUCKET"]

Even if there isn't a fix for this exact issue, is there some workaround? Having to make our boxes public is not feasible.

Thank you.

stevenscg commented 5 years ago

@darrenob I can't tell if you are having the same problem that I am, but have you tried changing util.rb like I did? That has worked for me locally, but it's a pain for my team since the official distribution doesn't work for our boxes.

darrenob commented 5 years ago

Thanks @stevenscg. In the end I moved the Vagrant boxes to an s3 bucket in the same account as the IAM user running vagrant up and that worked around the issue.

I didn't try the change to util.rb as it was not feasible to roll that out to my team. I guess it would have worked.

I will keep an eye on this thread as cross-account access using vagrant-s3auth seems like a core requirement. If it ever comes back I will definitely make use of it.

smaruy30 commented 4 years ago

@stevenscg I tried your possible solution but it did not work on my machine

      def self.s3_client(region = DEFAULT_REGION)
        ::Aws::S3::Client.new(region: region, force_path_style: true)
      end

@darrenob then, I made a little modification on util.rb which set AWS_REGION environment variable to DEFAULT_REGION(line 20)

$ less -N ~/.vagrant.d/gems/2.4.9/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/util.rb
      1 require 'aws-sdk'
      2 require 'log4r'
      3 require 'net/http'
      4 require 'uri'
      5
      6 module VagrantPlugins
      7   module S3Auth
      8     module Util
      9       S3_HOST_MATCHER = /^((?<bucket>[[:alnum:]\-\.]+).)?s3([[:alnum:]\-\.]+)?\.amazonaws\.com$/
     10
     11       # The list of environment variables that the AWS Ruby SDK searches
     12       # for access keys. Sadly, there's no better way to determine which
     13       # environment variable the Ruby SDK is using without mirroring the
     14       # logic ourself.
     15       #
     16       # See: https://github.com/aws/aws-sdk-ruby/blob/ab0eb18d0ce0a515254e207dae772864c34b048d/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L42
     17       AWS_ACCESS_KEY_ENV_VARS = %w(AWS_ACCESS_KEY_ID AMAZON_ACCESS_KEY_ID AWS_ACCESS_KEY).freeze
     18
     19       #DEFAULT_REGION = 'us-east-1'.freeze
     20       DEFAULT_REGION = ENV['AWS_REGION'].freeze
     21
     22       LOCATION_TO_REGION = Hash.new { |_, key| key }.merge(
     23         '' => DEFAULT_REGION,
     24         'EU' => 'eu-west-1'
     25       )

then it seems working

$ AWS_PROFILE=<<my-profile>> AWS_REGION=ap-northeast-1 vagrant box add <<my-box>> s3://<<bucket>>/package.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box '<<my-box>>' (v0) for provider:
    box: Downloading: s3://<<bucket>>/package.box
==> box: Box download is resuming from prior download progress
    box: Signing S3 request with key <<key>> loaded from profile <<my-profile>>
    box: Progress: 0% (Rate: 333k/s, Estimated time remaining: 1:02:12)
^C==> box: Waiting for cleanup before exiting...
==> box: Box download was interrupted. Exiting.

I am not sure if this modification is okay against the vagrant-s3auth specification( I do not code ruby often ), but the hard coded region("us-east-1") might be the problem

if the plugin absolutely needs 'us-east-1' as default region value, it could be nice if we could override DEFAULT_REGION with AWS_REGION.

cheers

WhoopInc / vagrant-s3auth

Request for box's Amazon S3 region was denied #38