Regional variation in getSignedUrl behaviour

plumdog commented 1 year ago

Describe the bug

There seem to be two kinds of region, that return different styles of signed URLs that have different behaviour.

This difference is demonstrated by the structure of the URL returned by eg:

await s3.getSignedUrlPromise("putObject", {
    Bucket: "my-bucket",
    Key: "test.txt",
    ContentType: "text/plain",
});

Some regions appear to be "strict" in some sense, and return URLs like (shown with newlines between params for clarity):

https://my-bucket.s3.ap-southeast-2.amazonaws.com/test.txt?
AWSAccessKeyId=...&
Content-Type=text%2Fplain&
Expires=...&
Signature=...&
x-amz-security-token=...

Some appear to be "permissive" and return URLs like:

https://mybucket.s3.ap-northeast-2.amazonaws.com/test.txt?
Content-Type=text%2Fplain&
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=...&
X-Amz-Date=...&
X-Amz-Expires=900&
X-Amz-Security-Token=...&
X-Amz-Signature=...&
X-Amz-SignedHeaders=host

In particular, the "strict" regions appear to allow a content type to be "baked in" to the signed URL, and throw an error if the upload doesn't set the same content type.

We found this change in behaviour because some application code that worked in a "permissive" region got deployed to a "strict" region and we started getting confusing signature errors.

My experimentation shows the two categories of region are:

Strict regions

ap-northeast-1
ap-southeast-1
ap-southeast-2
eu-west-1
sa-east-1
us-east-1
us-west-1
us-west-2

Permissive regions

ap-northeast-2
ap-northeast-3
ap-south-1
ca-central-1
eu-central-1
eu-north-1
eu-west-2
eu-west-3
us-east-2

Expected Behavior

Same behaviour across regions. At least, that the default options in the SDK result in the same behaviour across regions.

Current Behavior

Differing behaviour across regions.

Reproduction Steps

$ echo test > test.txt

# Permissive region, so works
$ export AWS_REGION=eu-west-2
$ export BUCKET_NAME=my-bucket-london
$ url=$(node -e 'const AWS = require("aws-sdk"); const s3 = new AWS.S3(); s3.getSignedUrlPromise("putObject", {Bucket: process.env.BUCKET_NAME, Key: "test.txt", ContentType: "application/json"}).then(console.log)')
$ curl -X PUT -H 'Content-Type: text/plain' --upload-file ./test.txt "$url"

# Strict region, so fails
$ export AWS_REGION=eu-west-1
$ export BUCKET_NAME=my-bucket-ireland
$ url=$(node -e 'const AWS = require("aws-sdk"); const s3 = new AWS.S3(); s3.getSignedUrlPromise("putObject", {Bucket: process.env.BUCKET_NAME, Key: "test.txt", ContentType: "application/json"}).then(console.log)')
$ curl -X PUT -H 'Content-Type: text/plain' --upload-file ./test.txt "$url"
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>...</AWSAccessKeyId>
<StringToSign>PUT

text/plain
1678720647
x-amz-security-token:...</StringToSign>
<SignatureProvided>...</SignatureProvided>
<StringToSignBytes>...</StringToSignBytes>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>

(Newlines added to error XML for legibility, and anything secret removed.)

Possible Solution

Change the defaults, perhaps in https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property, such that default behaviour is consistent across regions. Failing that, document this behaviour.

Additional Information/Context

I have not experimented with whether this is unique to the JS v2 SDK.

My next investigation, when I have time will be:

Does this affect other SDKs, eg the JS v3 SDK, or boto3?
Can I switch on enough logging to work out whether the SDK is making different initial requests to AWS for these different regions?

SDK version used

2.1333.0

Environment details (OS name and version, etc.)

n/a

plumdog commented 1 year ago

A quick check suggests that boto3 is not affected in the same way:

# London, "permissive" with JS SDK v2
$ export AWS_REGION=eu-west-2
$ export AWS_DEFAULT_REGION=eu-west-2
$ export BUCKET_NAME=my-bucket-london
$ url=$(python -c 'import boto3, os; s3 = boto3.client("s3"); print(s3.generate_presigned_url("put_object", Params=dict(Bucket=os.environ["BUCKET_NAME"], Key="test.txt", ContentType="application/json")))')
$ curl -X PUT -H 'content-type: text/plain' --upload-file ./test.txt "$url"
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>...</AWSAccessKeyId>
<StringToSign>...</StringToSign>
<SignatureProvided>...</SignatureProvided>
<StringToSignBytes>...</StringToSignBytes>
<CanonicalRequest>...</CanonicalRequest>
<CanonicalRequestBytes>...</CanonicalRequestBytes>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>

# Ireland, "strict" with JS SDK v2
$ export AWS_REGION=eu-west-1
$ export AWS_DEFAULT_REGION=eu-west-1
$ export BUCKET_NAME=my-bucket-ireland
$ url=$(python -c 'import boto3, os; s3 = boto3.client("s3"); print(s3.generate_presigned_url("put_object", Params=dict(Bucket=os.environ["BUCKET_NAME"], Key="test.txt", ContentType="application/json")))')
$ curl -X PUT -H 'content-type: text/plain' --upload-file ./test.txt "$url"
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>...</AWSAccessKeyId>
<StringToSign>PUT

text/plain
1678725007
x-amz-security-token:...</StringToSign>
<SignatureProvided>...</SignatureProvided>
<StringToSignBytes>...</StringToSignBytes>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>

However, there is still some slight regional variation here, as the structure of the error responses is different.

Digging further, and reviewing the structure of the URLs:

London:

https://my-bucket-london.s3.amazonaws.com/test.txt?
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=...&
X-Amz-Date=...&
X-Amz-Expires=3600&
X-Amz-SignedHeaders=content-type%3Bhost&
X-Amz-Security-Token=...&X-Amz-Signature=...

Ireland:

https://my-bucket-ireland.s3.amazonaws.com/test.txt?
AWSAccessKeyId=...&
Signature=...&
content-type=application%2Fjson&
x-amz-security-token=...&
Expires=...

So boto3 shows some regional variation in the implementation, but this appears not to impact behaviour.

plumdog commented 1 year ago

Think I have tracked this down to here: https://github.com/aws/aws-sdk-js/blob/master/lib/services/s3.js#L40-L46, so for regions that support older signing versions, they will be used for presigned URLs.

This, plus the fact that different signer versions:

have differing opinions about what should be "baked in" to the signed URL.

I think this is very surprising behaviour. It is sort of hinted at in the docs here https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property.

But eg https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#specify-signature-version says

For all AWS Regions, AWS SDKs use Signature Version 4 by default to authenticate requests.

which appears to not be the case here.

RanVaknin commented 1 year ago

Hey @plumdog ,

Great work on investigating this and providing a detailed report. This will help when we are root causing this. From a quick look it seems like you are correct and that some regions are causing the presigner to default to a different signature version.

I will investigate and let you know what we found!

Thanks, Ran~

RanVaknin commented 2 weeks ago

We are closing this issue since v2 is being put into maintenance mode. v3 uses sigv4 by default in all regions and this should not be an issue.

Thanks again for raising this. Ran~

aws / aws-sdk-js