search

WhoopInc / vagrant-s3auth

Vagrant plugin for private, versioned boxes on Amazon S3.
MIT License
108 stars 23 forks source link

Static Hosted Website S3 URLs are not correctly used #30

Closed lantrix closed 7 years ago

lantrix commented 7 years ago

Background

I'm using an S3 hosted private box & Metadata file. The S3 bucket is configured for static website hosting with the index document set to metadata.json.

I then have the following objects/keys in place

The metadata points to the box as shown:

{
    "name": "company/mybox",
    "description": "This box contains Ubuntu 14.04.5 LTS 64-bit.",
    "versions": [{
        "version": "0.1.0",
        "providers": [{
                "name": "virtualbox",
                "url": "s3://mybucket/myfolder/boxes/mybox_0.1.0.box",
                "checksum_type": "sha1",
                "checksum": "8a112b06a7f3c0c35b746e5ae7b7bcb8978eec54"
        }]
    }]
}

I can try to add the box via the s3 or http protocols and it fails using the s3 endpoint generated by the vagrant-s3auth plugin; or using the normal http/s endpoint:

S3 protocol

C:\VagrantTest> vagrant box add company/mybox "s3://mybucket/myfolder/"
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'company/mybox' (v0) for provider:
    box: Downloading: s3://mybucket/myfolder/
    box: Signing S3 request with key '**************' loaded from $AWS_ACCESS_KEY_ID
    box: Progress: 0% (Rate: 0/s, Estimated time remaining: --:--:--)
The "metadata.json" file for the box 'company/mybox' was not found.
Boxes require this file in order for Vagrant to determine the
provider it was made for. If you made the box, please add a
"metadata.json" file to it. If someone else made the box, please
notify the box creator that the box is corrupt.

http protocol

C:\VagrantTest> vagrant box add company/mybox "http://s3-ap-southeast-2.amazonaws.com/mybucket/myfolder/"
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'company/mybox' (v0) for provider:
    box: Downloading: http://s3-ap-southeast-2.amazonaws.com/mybucket/myfolder/
    box: Signing S3 request with key '****************' loaded from $AWS_ACCESS_KEY_ID
    box: Progress: 0% (Rate: 0curl:/s, Estimated time remaining: --:--:--)
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try again.

Vagrant debugging shows the plugin converting this to: https://s3-ap-southeast-2.amazonaws.com/mybucket/myfolder/?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential etc... Which returns a 404 not found

It DOES work if I point to the metadata.json URL directly; i.e. vagrant box add company/mybox "s3://mybucket/myfolder/metadata.json therefore there are no access or permission issues.

The reason for this seems to be that even though metadata.json has an object URI of https://s3-ap-southeast-2.amazonaws.com/mybucket/myfolder/metadata.json if you try to use https://s3-ap-southeast-2.amazonaws.com/mybucket/myfolder/ it will not work. This is noted in this stackoverflow answer that you need to use the aws s3 website endpoint for the index documents to work.

Digging around the AWS Documentation, and they document the following:

When you configure a bucket for website hosting, the website is available via the region-specific website endpoint. Website endpoints are different from the endpoints where you send REST API requests.

They also note:

From the S3 Management Console, selecting the target bucket, then Properties, then Static Website Hosting - it will show the website endpoint, for example mybucket.s3-website-ap-southeast-2.amazonaws.com

Issue

Now we come to the actual issue :) with the plugin (sorry for the long winded background).

Using the http protocol for the website endpoint still doesn't work:

C:\VagrantTest> vagrant box add mycompany/mybox "http://mybucket.s3-website-ap-southeast-2.amazonaws.com/myfolder/"
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'mycompany/mybox' (v0) for provider:
    box: Downloading: http://mybucket.s3-website-ap-southeast-2.amazonaws.com/myfolder/
    box: Signing S3 request with key '***************' loaded from $AWS_ACCESS_KEY_ID
    box: Progress: 0% (Rate: 0/s, Estimated time remaining: --:--:--)
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

The requested URL returned error: 404 Not Found

Enabling debug shows why.

DEBUG cli: Invoking command class: VagrantPlugins::CommandBox::Command::Root ["add", "mycompany/mybox", "http://mybucket.s3-website-ap-southeast-2.amazonaws.com/myfolder/"]
...
 INFO downloader: s3auth: Discovered S3 URL: http://mybucket.s3-website-ap-southeast-2.amazonaws.com/myfolder/
DEBUG downloader: s3auth: Bucket: "mybucket"
DEBUG downloader: s3auth: Key: "myfolder"
 INFO downloader: s3auth: Generating signed URL for GET
 INFO interface: detail: Signing S3 request with key '************' loaded from $AWS_ACCESS_KEY_ID
 INFO interface: detail:     box: Signing S3 request with key '************' loaded from $AWS_ACCESS_KEY_ID
    box: Signing S3 request with key '************' loaded from $AWS_ACCESS_KEY_ID
 INFO subprocess: Starting process: ["C:\\HashiCorp\\Vagrant\\embedded\\bin/curl.EXE", "-q", "--fail", "--location", "--max-redirs", "10", "--user-agent", "Vagrant/1.8.5 (+https://www.vagrantup.com; ruby2.2.3)", "--continue-at", "-", "--output", "C:/Users/lantrix/.vagrant.d/tmp/boxec029ce5c2b44aed1f4d3a1e785b7544e50234df", "https://s3-ap-southeast-2.amazonaws.com/mybucket/myfolder?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx"]
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  curl: (22) The requested URL returned error: 404 Not Found

The s3auth plugin is reconstructing from the website endpoint on http to the api endpoint on https which doesn't support index documents or redirect according to Amazon.

lantrix commented 7 years ago

I can't easily see a way using the SDK to generate the website endpoint; and apparently you can't https://github.com/aws/aws-sdk-ruby/issues/915#issuecomment-136775467

benesch commented 7 years ago

So using a website endpoint requires that your objects be publicly available, which rather defeats the purpose of this plugin, which is to support private, versioned boxes. So if you want to go that route, you don't need vagrant-s3auth at all. Does using the website URL http://mybucket.s3-website-ap-southeast-2.amazonaws.com/myfolder/ directly work if you uninstall vagrant-s3auth? If so, that's a definite bug. It seems our S3 URL matching is too aggressive and picks up the website URL. But the point is you don't need this plugin!

If you do want private boxes, you'll need to point directly to the metadata JSON. As you've discovered, that approach does work. If you don't like the ugliness of specifying s3://bucket/folder/metadata.json, make your metadata JSON a file without an extension, like s3://bucket/folder/box. If you go this route, see the caveat in the README about ensuring this file is served with Content-Type: application/json.

lantrix commented 7 years ago

Thanks for the response @benesch.

It doesn't work without the plugin as it is 403 forbidden as expected. Due to the fact that I haven't enabled public access on the S3 bucket; but instead was hoping to get access with signed requests from IAM users.

I've just realised my incorrect assumption of using IAM and the Website endpoint:

In order for your customers to access content at the website endpoint, you must make all your content publicly readable.

I do want private boxes, and I have pointed to the metadata directly. The ugliness is OK, and I see this is not an issue at all.

Cheers.

benesch commented 7 years ago

No problem. Enjoy!