SigV4 presigned URL fail always on S3

[michele-comitini]

Describe the bug Cannot create valid presigned url for S3 while using V4 signature. Same happens with aws s3 presign <url>

Steps to reproduce

my_config = Config(
    region_name = region,
    signature_version = 'v4',
)

S3_CLIENT = boto3.client(
    "s3",
    config=my_config,
)

signedurl = S3_CLIENT.generate_presigned_url(
        ClientMethod="put_object",
        Params={
            "Bucket": BUCKET,
            "Key": object_path,
            "Expires": 300,
            "ACL": "public-read",
            "ContentType": "image/jpeg",
        },
        HttpMethod="PUT",
    )

trying to upload to resulting URL gives error:

<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>EXAMPLEKEYEXAMPLE</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20210413T130851Z\n20210413/eu-west-1/s3/aws4_request\n94175ce956341268e890d7c27b6717141086fdfa3fdf1ff66734e3f958580562</StringToSign><SignatureProvided>5a2d0452c1902051aac533be1041bef43e2dc92f0bb0ee016250044c988ab538</SignatureProvided>

removing signature_version from my_config iif s3 endpoint supports v2 creates a valid signed URL

Expected behavior A valid url Debug logs N.A.

[kdaily]

Hi @michele-comitini,

I'm sorry to hear you're having an issue. I was not able to reproduce this with the most recent version of boto3 or the AWS CLI.

In general, signing issues are common with the scenarios described here: https://docs.aws.amazon.com/general/latest/gr/signature-v4-troubleshooting.html

If you provide more details, I might be able to narrow this down further:

1. complete debug logs
2. the version you're using
3. the region
4. the filename
5. what a successful event looks like (debug and resulting URL)

[kdaily]

You do have a typo in your value for signature_version - it should be s3v4 instead of v4.

[michele-comitini]

You do have a typo in your value for signature_version - it should be s3v4 instead of v4.

@kdaily thanks for investigating! Yet I am not out of it!

Here is what I have done:

• create a new access key. The old one was quite old
• create a new bucket (eu-south-1 region)
• v4 -> s3v4

The OS is Linux 64 bit, python3.8.

boto3 version: 1.17.49 botocore version: 1.20.50

N.B. I get same result running the same code as Lambda or trying to get signed URL by aws cli

rewrote the script:

# -*- coding: utf-8 -*-

import os
import boto3
import botocore
import logging

import sys

import requests

from botocore.client import Config

logger = logging.getLogger()
logger.setLevel(logging.INFO)

region = "eu-south-1"

my_config = Config(
    region_name=region,
    signature_version="s3v4",
)
profile = os.getenv("AWS_PROFILE")
BUCKET = os.getenv("S3BUCKET")

S3_CLIENT = boto3.client(
    "s3",
    config=my_config,
)

photo_file_path = sys.argv[1]

photo_file_name = os.path.basename(photo_file_path)

signedurl = S3_CLIENT.generate_presigned_url(
    ClientMethod="put_object",
    Params={
        "Bucket": BUCKET,
        "Key": photo_file_name,
        "Expires": 300,
        "ACL": "public-read",
        "ContentType": "image/jpeg",
    },
    HttpMethod="PUT",
)

print(f"boto3 version: {boto3.__version__}")
print(f"botocore version: {botocore.__version__}")

print(signedurl)
with open(f"{photo_file_path}", "rb") as gattofile:
    respo = requests.put(signedurl, headers={"Content-Type": "image/jpeg"}, data=gattofile)
    print(respo.content)

output (hopefully) without sensitive data including boto3 + botocore version:

boto3 version: 1.17.49
botocore version: 1.20.50
https://s3.eu-south-1.amazonaws.com/xxxxxxxxxxxxxx/gatto.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYXXXXXXXXXXXXX%2F20210420%2Feu-south-1%2Fs3%2Faws4_request&X-Amz-Date=20210420T095030Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bexpires%3Bhost%3Bx-amz-acl&X-Amz-Signature=96ccafdb1ad0569d8164b68f44aed6bd87f75ac4e637058114692f3ff03bcbac
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAYXXXXXXXXXXXXX</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20210420T095030Z\n20210420/eu-south-1/s3/aws4_request\neaa61ac0f06a93f665ffc7c402ab1fb3de9a0063ba3f8430970cadbae527d7c2</StringToSign><SignatureProvided>96ccafdb1ad0569d8164b68f44aed6bd87f75ac4e637058114692f3ff03bcbac</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 31 30 34 32 30 54 30 39 35 30 33 30 5a 0a 32 30 32 31 30 34 32 30 2f 65 75 2d 73 6f 75 74 68 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 65 61 61 36 31 61 63 30 66 30 36 61 39 33 66 36 36 35 66 66 63 37 63 34 30 32 61 62 31 66 62 33 64 65 39 61 30 30 36 33 62 61 33 66 38 34 33 30 39 37 30 63 61 64 62 61 65 35 32 37 64 37 63 32</StringToSignBytes><CanonicalRequest>PUT\n/xxxxxxxxxxxxxx/gatto.jpg\nX-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAYXXXXXXXXXXXXX%2F20210420%2Feu-south-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20210420T095030Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=content-type%3Bexpires%3Bhost%3Bx-amz-acl\ncontent-type:image/jpeg\nexpires:\nhost:s3.eu-south-1.amazonaws.com\nx-amz-acl:\n\ncontent-type;expires;host;x-amz-acl\nUNSIGNED-PAYLOAD</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 67 61 74 74 69 2e 6d 69 6c 61 6e 6f 2e 69 6d 61 67 65 73 2f 67 61 74 74 6f 2e 6a 70 67 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 59 34 51 32 56 33 4e 5a 36 4c 5a 4e 51 43 4f 4a 25 32 46 32 30 32 31 30 34 32 30 25 32 46 65 75 2d 73 6f 75 74 68 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 31 30 34 32 30 54 30 39 35 30 33 30 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 63 6f 6e 74 65 6e 74 2d 74 79 70 65 25 33 42 65 78 70 69 72 65 73 25 33 42 68 6f 73 74 25 33 42 78 2d 61 6d 7a 2d 61 63 6c 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 69 6d 61 67 65 2f 6a 70 65 67 0a 65 78 70 69 72 65 73 3a 0a 68 6f 73 74 3a 73 33 2e 65 75 2d 73 6f 75 74 68 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 0a 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 65 78 70 69 72 65 73 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 61 63 6c 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes><RequestId>CMA6V2TF5EW17E12</RequestId><HostId>Ab4q2yYUEEgnHw21No0KhwaqXpiOd5ClHT4MmAmBx9a4W8Za7AcR2prU3HNSNT5jmhx4+e0y4sc=</HostId></Error>'

[michele-comitini]

Gist with some logs: https://gist.github.com/michele-comitini/fecc94e6fc87f071106aae79b66f0aa0

[JorisLA]

Got the same error with greengrass (reset_deployments api)

The OS is Linux 64 bit, python3.7.5

boto3==1.17.54 botocore==1.20.54

client = boto3.client(
    'greengrass',
    aws_access_key_id=config.aws_access_key_id,
    aws_secret_access_key=config.aws_secret_access_key,
    region_name=config.aws_default_region
)

reset_response = client.reset_deployments(Force=True, GroupId=group_id)

error :

botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the ResetDeployments operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/greengrass/groups/88d82b49-65a0-4b0a-a341-e238e22f4e86/deployments/%24reset

host:greengrass.eu-west-1.amazonaws.com
x-amz-date:20210420T131121Z

host;x-amz-date
e2c561fddf9afd1986904a4fe67cda75395efe3f30ba6cfb47bc888c7cd5de42'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20210420T131121Z
20210420/eu-west-1/greengrass/aws4_request
ba7b731ab265895ee07216aaf8ca67e816cdb377e8dd13cf8d4581cf451147b8'

It seems the error comes just after 1.20.52 botocore version.

When using CrtSigV4Auth class I got the error but with SigV4Auth it works fine. To switch from one to the other I export BOTO_DISABLE_CRT to true

PS: It works also fine with aws cli

aws --version
aws-cli/1.19.54 Python/2.7.18 Linux/5.4.0-48-generic botocore/1.20.54

aws greengrass reset-deployments \
    --group-id "ad9d7e24-17dc-4d81-ba66-3619bb06645c" \
    --force

[nateprewitt]

Hey @JorisLA,

Thanks for that report. I believe that's a separate issue but definitely looks like something that needs to be fixed. Would you mind opening a new issue with your post above so we can track it separately?

[nateprewitt]

Hi @michele-comitini,

To follow up on your issue, it looks like this is a case of some slight misconfigurations. We took the sample code you provided and found the request being sent is missing some of the required headers you're specifying. There's also what appears to be an unintentional use of Expires instead of ExpiresIn.

When you perform a request to a presigned URL, it expects any header arguments you supplied while building it will be present in the request. If they aren't present, or don't match, the request will fail. In your case, you've specified Params (ACL, ContentType, and Expires) which translate to x-amz-acl, content-type, and expires. There's also an implicit required host header, but Requests and most browsers include this for you. You can see the full requirements for your request in the presigned URL itself here: X-Amz-SignedHeaders=content-type%3Bexpires%3Bhost%3Bx-amz-acl.

In regards to the Expires parameter though, this is documented as a datetime object, not an integer. So when you specify 300, it's not 300 seconds, it's epoch + 300 (Thursday, January 1, 1970 12:05:00 am UTC). The argument you're likely looking for is ExpiresIn which is provided to the generate_presigned_url function rather than as a Param.

To fix both of those issues, you'll want to add those headers into your request and update the ExpiresIn usage.

e.g.

[...]

signedurl = S3_CLIENT.generate_presigned_url(
    ClientMethod="put_object",
    Params={
        "Bucket": BUCKET,
        "Key": photo_file_name,
        "ACL": "public-read",
        "ContentType": "image/jpeg",
    },
    HttpMethod="PUT",
    ExpiresIn=300,
)

[...]

headers = {
    "Content-Type": "image/jpeg",
    "X-Amz-ACL": "public-read"
}
with open(f"{photo_file_path}", "rb") as gattofile:
    respo = requests.put(signedurl, headers=headers, data=gattofile)
    print(respo.content)

[michele-comitini]

@kdaily thank you very much! I confirm that it works, if I correct and add proper headers. I was rather confused by the error message: I wish it could give more hints on the missing parts

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.