Difference in SigV4 calculation between S3 and botocore?

[ahmayun]

Describe the bug

In some instances there seems to be a difference between how botocore constructs the canonical request and how S3 constructs it. Specifically, botocore seems to normalize the request URI whereas the server does not.

When the code provided in the reproduction steps section is run, a 403: SignatureDoesNotMatch response is received. I believe the issue is because the resource URI is different. The % signs in the url are escaped by the library but not by the server.

Here is the canonical request constructed by the library:

GET
/fuzzingbucket/%25C3%2582%25F1%25B2%25A7%25BF%25C2%25BB0

host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240627T191628Z
x-amz-security-token:<redacted>

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

This is the canonical request constructed by the server:

GET
/fuzzingbucket/%C3%82%F1%B2%A7%BF%C2%BB0

host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240627T191628Z
x-amz-security-token:<redacted>

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

I got the canonical request being constructed by the library by adding a print statement under this line

Expected Behavior

The signatures should match.

Current Behavior

Signatures do not match

Reproduction Steps

Run the following code after adding your credentials, should be self contained:

import boto3
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
import requests

session = boto3.Session(
        aws_access_key_id='<secret key id>',
        aws_secret_access_key='<secret key>',
        aws_session_token='<your session token>'
)
credentials = session.get_credentials()
creds = credentials.get_frozen_credentials()

headers = {}
headers['X-Amz-Content-SHA256'] = 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' # sha256 of an empty string
request = AWSRequest(method="GET", url="https://s3.amazonaws.com/fuzzingbucket/%C3%82%F1%B2%A7%BF%C2%BB0", headers=headers)

SigV4Auth(creds, "s3", 'us-east-1').add_auth(request)
prepped = request.prepare()

response = requests.get(prepped.url, headers=prepped.headers)
print(response.text)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

python botocore library version: 1.34.133

Environment details (OS name and version, etc.)

Darwin 80a99729a79c 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:25 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6030 arm64

[nateprewitt]

Hi @ahmayun,

The signers in Botocore are not intended to be used in external code. They're an internal implementation detail and we don't generally support use cases like this. The current issue you're hitting is you're using the wrong signer. SigV4Auth is for all non-S3 AWS services. S3 has its own variant of SigV4 that's implemented in S3SigV4Auth.

We do have a project in very early phases of development in https://github.com/awslabs/aws-sdk-python-signers which is intended to be our longer term option for signing.

[ahmayun]

Thank you for the response @nateprewitt! I will check out the project you mentioned.

I admit my use case is a bit strange, I need to sign some arbitrary requests. It works with the S3SigV4Auth!