Support for other S3 hosts / regions

[stevenscg]

Thanks for a great plugin. Can it be used for a bucket outside of US Standard?

When using an S3 bucket outside of US Standard like this:

https://s3-us-west-2.amazonaws.com/bucket/path/to/file.box

I am receiving a 403 Forbidden error when running vagrant up:

The requested URL returned error: 403 Forbidden
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

I have changed S3_HOST lib/vagrant-s3auth.rb to match my region:

S3_HOST         = 's3-us-west-2.amazonaws.com'

When running vagrant up with debug, I was able see the Signing URL for S3 box message and capture the signed URL. Entering that into a browser within the expiry time seemed to work.

Is there something else that needs to be adjusted for an S3 region besides US Standard?

[stevenscg]

I just reinstalled the plugin and tried downloading a box from the US standard region. Same 403 Forbidden error. I was still able to manually download the box via browser.

This is all with Vagrant 1.6.2.

[benesch]

Glad you're getting some use out of it! S3's special URL rules for US standard buckets causes all sorts of problems; I'll look into it ASAP.

[stevenscg]

Indeed. I'm not a rubyist, but around to help if you need it.

It sure seems like the signing works with my change to S3_HOST since I can download it from the browser. It could be a quirk in recent Vagrant release(s).

[kmontag]

FWIW, non-standard buckets are working fine for me with a hack like VagrantPlugins::S3Auth::S3_HOST = 's3-us-west-1.amazonaws.com' in my Vagrantfile. Using Vagrant 1.6.2 as well. Seems like maybe all that's needed is to use a regex for the host matching in s3_url?.

[stevenscg]

Thanks @kmontag. Very slick, though I did get some "already defined" errors when I put that in my Vagrantfile.

[stevenscg]

@benesch

I also tried the plugin against a US Standard bucket with Vagrant 1.6.3 and still get the 403 errors. Debug output below.

The last signed URL I see is the HEAD call. I am also using credentials from an IAM user with a policy for this bucket, but I don't think that's an issue since the url works fine with the browser if I load it before expiration.

 INFO runner: Running action: #<Vagrant::Action::Builder:0x00000100c859a0>
 INFO warden: Calling IN action: #<Vagrant::Action::Builtin::BoxAdd:0x00000100c96d40>
 INFO box_add: Not transforming non-S3 box: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box
 INFO environment: Running hook: authenticate_box_url
 INFO runner: Preparing hooks for middleware sequence...
 INFO runner: 3 hooks defined.
 INFO runner: Running action: #<Vagrant::Action::Builder:0x00000100d54b38>
 INFO warden: Calling IN action: #<VagrantPlugins::Login::ActionAuthenticateBox:0x00000101351018>
 INFO warden: Calling IN action: #<VagrantPlugins::S3Auth::Action::AuthenticateBoxUrl:0x00000101350ff0>
 INFO authenticate_box_url: Signing URL for S3 box: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box
 INFO warden: Calling OUT action: #<VagrantPlugins::S3Auth::Action::AuthenticateBoxUrl:0x00000101350ff0>
 INFO warden: Calling OUT action: #<VagrantPlugins::Login::ActionAuthenticateBox:0x00000101351018>
 INFO box_add: Downloading box: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box?AWSAccessKeyId=MYAWSKEY&Expires=1402493529&Signature=X01E8unsGgOULap%2BEEYyOM432M8%3D => /Volumes/Users/myuser/.vagrant.d/tmp/boxfebd611830309aa362b6f7cc39e211fa8d75941f
 INFO downloader: HEAD: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box?AWSAccessKeyId=MYAWSKEY&Expires=1402493529&Signature=X01E8unsGgOULap%2BEEYyOM432M8%3D
 INFO subprocess: Starting process: ["/Applications/Vagrant/bin/../embedded/bin/curl", "-I", "--fail", "--location", "--max-redirs", "10", "--user-agent", "Vagrant/1.6.3", "--continue-at", "-", "-H", "Accept: application/json", "https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box?AWSAccessKeyId=MYAWSKEY&Expires=1402493529&Signature=X01E8unsGgOULap%2BEEYyOM432M8%3D"]
 INFO subprocess: Command in the installer. Specifying DYLD_LIBRARY_PATH...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG subprocess: stderr: curl: (22) The requested URL returned error: 403 Forbidden
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 22
 WARN downloader: Downloader exit code: 22
 INFO interface: output: Adding box 'centos-6.5-x86-64' (v0) for provider: virtualbox
 INFO interface: output: ==> default: Adding box 'centos-6.5-x86-64' (v0) for provider: virtualbox
==> default: Adding box 'centos-6.5-x86-64' (v0) for provider: virtualbox
 INFO box_add: Downloading box: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box => /Volumes/Users/myuser/.vagrant.d/tmp/box2e4609f8412cc5cf2434282689b78c80ff3a1b74
 INFO interface: detail: Downloading: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box
    default: Downloading: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box
 INFO downloader: Downloader starting download: 
 INFO downloader:   -- Source: https://s3.amazonaws.com/bucket/centos-6.5-x86-64-1401991132.box
...
(22) The requested URL returned error: 403 Forbidden
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 31999
DEBUG subprocess: Exit status: 22
 WARN downloader: Downloader exit code: 22

[benesch]

Very strange. I'm using it successfully on a 1.6.3 install at the moment.

I'm going to add an explicit region option so you don't have to hack the plugin, then get started on some integration tests to see if I can reproduce.

[benesch]

@kmontag are you by chance using a metadata box?

[kmontag]

@benesch yes I am. The metadata is local if that makes any difference.

Also worth noting that someone else on my team (using the same Vagrantfile with the S3_HOST hack, the same box version, and the same AWS credentials) is getting 403s when trying to download the box, and VAGRANT_LOG=info suggests that the S3 URL isn't being transformed at all on their system. No idea what the difference is between our systems, unfortunately.

[benesch]

I've tracked down that if you set box_url to a simple box instead of a metadata box, Vagrant calls authenticate_box_url, but then downloads from the non-authenticated URL. :frowning: @stevenscg, this explains your problem.

@kmontag is your teammate using the direct S3 url?

I'll start working on a fix. It's a substantial refactor, though, and requires hooking into the curl downloader layer instead of the box_add layer.

[kmontag]

@benesch no, my teammate is also using the local metadata box. I'll see if I can figure out some way to reproduce that problem consistently.

[benesch]

I just cut a new prerelease

vagrant plugin install vagrant-s3auth --plugin-version '0.1.0.alpha1'

that implements the fixes mentioned above. Working for me on both metadata and simple boxes. @kmontag and @stevenscg please give it a shot!

If it's stable for you guys, I'll release as 0.1.0. A couple specs/integration tests should get us to 1.0.

[stevenscg]

@benesch: That alpha1 release worked perfectly with my .box file in S3 us-west-2. Nice work!

[benesch]

Sweet! Released as v0.1.0.

@kmontag does this fix your teammate's issue?

[kmontag]

Sorry for the slow followup - yes, looks like it's working on his machine as well :) Thanks!

On Fri, Jun 13, 2014 at 11:29 AM, Nikhil Benesch notifications@github.com wrote:

Sweet! Released as v0.1.0.

@kmontag https://github.com/kmontag does this fix your teammate's issue?

— Reply to this email directly or view it on GitHub https://github.com/WhoopInc/vagrant-s3auth/issues/1#issuecomment-46044724 .

[phantomwhale]

Also had region issues, fixed with overwriting the constant for now, but would be nice to get rid of the warning every time I invoke vagrant now.

VagrantPlugins::S3Auth::S3_HOST = 's3-ap-southeast-1.amazonaws.com'
config.vm.box_url = "s3://my-bucket-name/vagrant/ubuntu-14.04.1-server-amd64_virtualbox.box"

[debo]

@benesch does the bucket need to have a specific name? In my case the downloading URL doesn't have any credentials appended so that I keep hitting a 403. Ideas?

@stevenscg you might want to remove your AWS credentials from your comments ;-)

[stevenscg]

@MarcoDeBortoli Thanks for the heads up. I see dummy credentials in there now. Did you clean em out already?

[debo]

@stevenscg nope, maybe @benesch did or someone in the Whoopinc team.

[benesch]

1.0 has landed! This should finally be fixed. Either specify the region in the box_url

http://s3-eu-west-1.amazonaws.com/bucket/key

or use the shorthand, which auto-determines the region:

s3://bucket/key

See the README for details!