S3 CORS issue when setting up datasource

[bdadam]

The browser does not set the Origin header when setting up an S3 source. Therefore the API request to S3 fails with the following error:

Access to XMLHttpRequest at 'https://<bucketname>.s3.amazonaws.com//temp-to-be-deleted-76aghsdhamj2lk3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<REDACTED>%2F20210912%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210912T223840Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&list-type=2&max-keys=750&X-Amz-Signature=<REDACTED>' from origin 'https://diffuse.sh' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

According to the AWS docs S3 only treats requests as corss-origin request if the Origin header is present.

Here is the request copied from Chrome dev tools.

curl 'https://<bucktname>.s3.amazonaws.com/<bucketname>?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<REDACTED>%2F20210912%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210912T223840Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&list-type=2&max-keys=750&X-Amz-Signature=<REDACTED>' \
  -H 'Referer: https://diffuse.sh/' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36' \
  -H 'DNT: 1' \
  --compressed

Browser: Chromium Version 93.0.4577.63

Possible cause: missing mode: 'cors' on the fetch(...) call?

[icidasset]

Hi 👋 Have you followed the CORS instructions? https://nightly.diffuse.sh/about/cors/#CORS__S3 You need to change these settings on your S3 bucket.

[bdadam]

@icidasset Of course I did. ~Actually the instructions are out of date (the AWS console only supports JSON format now) so I used my own CORS config. But this is for a separate issue or PR.~ (Sorry, this is solved now.)

The issue is that S3 only sends the Access-Control-Allow-Origin header if there is an Origin header in the request. Which is missing - that's why I assume the mode is missing from the fetch call.

The linked S3 docs page says:

Verify that the Origin header in your request matches at least one of the AllowedOrigin elements in the specified CORSRule. But in this case there was no Origin header in the request.

[bdadam]

Oh sorry, it's in JSON format now in the nightly builds. But I used exactly that configuration.

[icidasset]

Just checking 😉 Hmm ok, my S3 buckets do work. Can you try out the demo source? You can add that by clicking "insert demo" on the tracks screen when you have no sources. That's an S3 bucket as well.

Also just in case, can you tell me what browser (& browser mode) and OS you're using?

Thanks!

[bdadam]

Thanks @icidasset for taking the time.

The demo source works fine.

I re-did everything from the beginning. I still got the same error. But now I may have some clues what's going wrong.

I noticed that the create source page asks for the s3 region but it doesn't put the region into the URL domain. The request goes to https://<bucketname>.s3.amazonaws.com/?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210914T221721Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&list-type=2&max-keys=500&X-Amz-Signature=...

If I copy the request as cURL from the dev-tools and execute the command in terminal I get a <Error><Code>TemporaryRedirect</Code> error. According to AWS this could be because the bucket was created less than 24 hours ago. Which is true I only created this bucket 10 minutes ago for testing.

AWS suggest the following:

If you're using an Amazon CloudFront distribution with an Amazon S3 origin, CloudFront forwards requests to the default S3 endpoint ( s3.amazonaws.com). The default S3 endpoint is in the us-east-1 Region. If you must access Amazon S3 within the first 24 hours of creating the bucket, you can change the origin domain name of the distribution. The domain name must include the Regional endpoint of the bucket. For example, if the bucket is in us-west-2, you can change the origin domain name from awsexamplebucketname.s3.amazonaws.com to awsexamplebucket.s3.us-west-2.amazonaws.com.

Then I created a bucket in the us-east-1 region. And then it worked :tada: :)

My suggestion would be to always use the region specific S3 endpoint, e.g. https://<bucketname>.s3.eu-west-1.amazonaws.com. What do you think?

[icidasset]

Oh interesting, thanks for going through all that! I've added the region in the url as you suggested, which seems to work fine 👍

You can try it out on https://nightly.diffuse.sh/ You might need to refresh a couple of times to get the change (service workers and all that).

[icidasset]

Deployed fix to stable and released v3.