aldy120 / s3-note

Note for Amazon S3

0 stars 0 forks source link

SignatureDoesNotMatch #4

Open aldy120 opened 5 years ago

aldy120 commented 5 years ago

Credential Error

This error happens to me when I type a wrong credential when I use aws configure to set my secret access key.

Forward `Host` header to S3 REST Endpoint

If you forward the host header to S3 REST endpoint, you will see the following error in the page.

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<AWSAccessKeyId>AKIAJUCRW6H7RQOAY47A</AWSAccessKeyId>
<StringToSign>
AWS4-HMAC-SHA256 20190628T091700Z 20190628/ap-northeast-1/s3/aws4_request 8ad576b31ccc49e6cecf2deac912a8aaf89fd8d027e43a6716e0e8f27d3ecd27
</StringToSign>
<SignatureProvided>
4810917528bd35e30c14a4c22e5aaa622435fff586d184a7d02e2d9cb26e2408
</SignatureProvided>
<StringToSignBytes>
41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 39 30 36 32 38 54 30 39 31 37 30 30 5a 0a 32 30 31 39 30 36 32 38 2f 61 70 2d 6e 6f 72 74 68 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 38 61 64 35 37 36 62 33 31 63 63 63 34 39 65 36 63 65 63 66 32 64 65 61 63 39 31 32 61 38 61 61 66 38 39 66 64 38 64 30 32 37 65 34 33 61 36 37 31 36 65 30 65 38 66 32 37 64 33 65 63 64 32 37
</StringToSignBytes>
<CanonicalRequest>
GET /index.html host:dtkcceqegrp5x.cloudfront.net x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190628T091700Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
</CanonicalRequest>
<CanonicalRequestBytes>
47 45 54 0a 2f 69 6e 64 65 78 2e 68 74 6d 6c 0a 0a 68 6f 73 74 3a 64 74 6b 63 63 65 71 65 67 72 70 35 78 2e 63 6c 6f 75 64 66 72 6f 6e 74 2e 6e 65 74 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 39 30 36 32 38 54 30 39 31 37 30 30 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35
</CanonicalRequestBytes>
<RequestId>5526469A1F513640</RequestId>
<HostId>
zlIHYquPthZr0fHpO4QhVpIo4lMlJqaWJAQwYdSf3+mtuQoHPNtMMPHZ2Ka2CkL1yF0gaQ5mwwU=
</HostId>
</Error>

aldy120 commented 3 years ago

Two other scenarios.

S3 presigned URL with wrong HTTP method

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAUXOIYM7XYQRKV2OL</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20201001T061821Z
20201001/eu-central-1/s3/aws4_request
5239afc2ef6807bdb19ea7385f480397d5199479dc4a7b78e5ab22af96fbdb94</StringToSign><SignatureProvided>44ad9d09988e77cc69147fd4b5c66e97ffc0bea60e850628a4e55b08cea1b0b1</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 31 30 30 31 54 30 36 31 38 32 31 5a 0a 32 30 32 30 31 30 30 31 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 35 32 33 39 61 66 63 32 65 66 36 38 30 37 62 64 62 31 39 65 61 37 33 38 35 66 34 38 30 33 39 37 64 35 31 39 39 34 37 39 64 63 34 61 37 62 37 38 65 35 61 62 32 32 61 66 39 36 66 62 64 62 39 34</StringToSignBytes><CanonicalRequest>GET
/test-predigned.txt
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAUXOIYM7XYQRKV2OL%2F20201001%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20201001T061821Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=host
host:fra-bucket-12345678.s3.amazonaws.com

host
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
UNSIGNED-PAYLOAD</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 74 65 73 74 2d 70 72 65 64 69 67 6e 65 64 2e 74 78 74 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 55 58 4f 49 59 4d 37 58 59 51 52 4b 56 32 4f 4c 25 32 46 32 30 32 30 31 30 30 31 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 30 31 30 30 31 54 30 36 31 38 32 31 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 66 72 61 2d 62 75 63 6b 65 74 2d 31 32 33 34 35 36 37 38 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes><RequestId>520E6E4256E6DB07</RequestId><HostId>NlRZ94ZfvVIgGpTt+eZmYnYT3YabdaLJaWXmj1fyz2reOSDLmNvEjQ+79uNGP7EoSjnvrj5mcD4=</HostId></Error>

S3 presigned URL with Wrong secret_access_key

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAUXOIYM7XYQRKV2OL</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20201001T062808Z
20201001/eu-central-1/s3/aws4_request
6778a62568737cd078bf97220e6348040efbd7894d8d04b53ba83da4354bfa03</StringToSign><SignatureProvided>c569c6bd9e87c6d730b919d1674c833d256eec697385fe59c037f715a39018d4</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 31 30 30 31 54 30 36 32 38 30 38 5a 0a 32 30 32 30 31 30 30 31 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 36 37 37 38 61 36 32 35 36 38 37 33 37 63 64 30 37 38 62 66 39 37 32 32 30 65 36 33 34 38 30 34 30 65 66 62 64 37 38 39 34 64 38 64 30 34 62 35 33 62 61 38 33 64 61 34 33 35 34 62 66 61 30 33</StringToSignBytes><CanonicalRequest>PUT
/test-predigned.txt
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAUXOIYM7XYQRKV2OL%2F20201001%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20201001T062808Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=host
host:fra-bucket-12345678.s3.amazonaws.com

host
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
UNSIGNED-PAYLOAD</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 74 65 73 74 2d 70 72 65 64 69 67 6e 65 64 2e 74 78 74 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 55 58 4f 49 59 4d 37 58 59 51 52 4b 56 32 4f 4c 25 32 46 32 30 32 30 31 30 30 31 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 30 31 30 30 31 54 30 36 32 38 30 38 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 66 72 61 2d 62 75 63 6b 65 74 2d 31 32 33 34 35 36 37 38 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes><RequestId>49682B68B444BCD5</RequestId><HostId>6RUjvjuPyAUSxtZVx/su6L1HIsdduMjKmTYVs05kxvhUQctYNmTkPk5CDy+AFATz2xiHro5E/es=</HostId></Error>

aldy120 commented 3 years ago

簽名時沒有使用 use_accelerate_endpoint 相關參數。但最後用了 S3 Transfer Accelerate 的 domain

aldy120 commented 3 years ago

To disable the Expect header in Golang SDK https://github.com/aws/aws-sdk-go-v2/blob/v0.24.0/service/s3/api_client.go#L20-L67

aldy120 commented 3 years ago

https://github.com/aws/aws-sdk-java/issues/1919 If customer update to HttpClient 4.5.7 (updating Spring boot to 2.1.3 might cause it), the double slash // and plus sign + will be modify before sending. This cause the signature mismatch error.

aldy120 commented 3 years ago

Python SDK signed `content-type` as canonical header.

Generate put_object presigned url.

import boto3
url = boto3.client('s3').generate_presigned_url(
ClientMethod='put_object', 
Params={'Bucket': 'test-dub-12345678', 'Key': 'index.html'},
ExpiresIn=3600)

print(url)

https://test-dub-12345678.s3.amazonaws.com/index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK123L&Signature=DbaWfDYz2DbGv6zeW9BVj11oX%2Fs%3D&Expires=1627169559

Without `content-type`: success

curl -v -X PUT --data 123 -H 'Content-Type: ' 'https://test-dub-12345678.s3.amazonaws.com/index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK123L&Signature=DbaWfDYz2DbGv6zeW9BVj11oX%2Fs%3D&Expires=1627169559'
> PUT /index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK123L&Signature=DbaWfDYz2DbGv6zeW9BVj11oX%2Fs%3D&Expires=1627169559 HTTP/1.1
> Host: test-dub-12345678.s3.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 3
> 
* upload completely sent off: 3 out of 3 bytes
< HTTP/1.1 200 OK
< x-amz-id-2: 4Lkb2TJf+2u5W+SgdLFsOCEUol1EV7rWdRVMfk3Y0b8/HTriaZdigVWQcRRhEost7gg0zsDf738=
< x-amz-request-id: 930EP3BSJWPWJF0K
< Date: Sat, 24 Jul 2021 22:33:39 GMT
< ETag: "202cb962ac59075b964b07152d234b70"
< Server: AmazonS3
< Content-Length: 0
< 
* Connection #0 to host test-dub-12345678.s3.amazonaws.com left intact
* Closing connection 0

With `Content-Type: application/x-www-form-urlencoded`

The is set by curl by default.

curl -v -X PUT --data 123 'https://test-dub-12345678.s3.amazonaws.com/index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK123L&Signature=DbaWfDYz2DbGv6zeW9BVj11oX%2Fs%3D&Expires=1627169559'

> PUT /index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK123L&Signature=DbaWfDYz2DbGv6zeW9BVj11oX%2Fs%3D&Expires=1627169559 HTTP/1.1
> Host: test-dub-12345678.s3.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 3
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 3 out of 3 bytes
< HTTP/1.1 403 Forbidden
< x-amz-request-id: Q9KKAWHS0CEYJHWZ
< x-amz-id-2: Mt2DLWx14jsjzuA4P5CMVDYqiqbO+LuLSXDcM20KZ4zI63mAS4FI2jDB0PBbsbfkNQ09Y8Nfygg=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Sat, 24 Jul 2021 22:34:08 GMT
< Server: AmazonS3
< Connection: close
< 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAUXOIYM7XYQRK123L</AWSAccessKeyId><StringToSign>PUT

application/x-www-form-urlencoded
1627169559
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
/test-dub-12345678/index.html</StringToSign><SignatureProvided>DbaWfDYz2DbGv6zeW9BVj11oX/s=</SignatureProvided><StringToSignBytes>50 55 54 0a 0a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0a 31 36 32 37 31 36 39 35 35 39 0a 2f 74 65 73 74 2d 64 75 62 2d 31 32 33 34 35 36 37 38 2f 69 6e 64 65 78 2e 68 74 6d 6c</StringToSignBytes><RequestId>Q9KKAWHS0CEYJHWZ</RequestId><HostId>Mt2DLWx14jsjzuA4P5CMVDYqiqbO+LuLSXDcM20KZ4zI63mAS4FI2jDB0PBbsbfkNQ09Y8Nfygg=</HostId></Error>

Use wrong method (GET)

curl -v 'https://test-dub-12345678.s3.amazonaws.com/index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK1234&Signature=YDBwKxFzhmdlD%2BezBv6PfAMqORo%3D&Expires=1627174168' 

> GET /index.html?AWSAccessKeyId=AKIAUXOIYM7XYQRK1234&Signature=YDBwKxFzhmdlD%2BezBv6PfAMqORo%3D&Expires=1627174168 HTTP/1.1
> Host: test-dub-12345678.s3.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< x-amz-request-id: 1NN6CXSXCCARH4TX
< x-amz-id-2: pVndxLtFFNqfqOBgLSdt19xCqrHMcu5R0CPpcSY9hjTAfASQgame/AmSq8h5CgD9xz+0QNhDUmE=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Sat, 24 Jul 2021 23:49:49 GMT
< Server: AmazonS3
< 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAUXOIYM7XYQRK1234</AWSAccessKeyId><StringToSign>GET

1627174168
* Connection #0 to host test-dub-12345678.s3.amazonaws.com left intact
/test-dub-12345678/index.html</StringToSign><SignatureProvided>YDBwKxFzhmdlD+ezBv6PfAMqORo=</SignatureProvided><StringToSignBytes>47 45 54 0a 0a 0a 31 36 32 37 31 37 34 31 36 38 0a 2f 74 65 73 74 2d 64 75 62 2d 31 32 33 34 35 36 37 38 2f 69 6e 64 65 78 2e 68 74 6d 6c</StringToSignBytes><RequestId>1NN6CXSXCCARH4TX</RequestId><HostId>pVndxLtFFNqfqOBgLSdt19xCqrHMcu5R0CPpcSY9hjTAfASQgame/AmSq8h5CgD9xz+0QNhDUmE=</HostId></Error>

aldy120 commented 2 years ago

Python SDK (boto3)

Note:

signature_version must be s3v4.
For the region that doesn't support global S3 endpoint, add 'addressing_style': 'virtual' to S3 settings. This can prevent the URL like xxx.s3.amazonaws.com and 307 issues.

import boto3
from botocore.config import Config

my_config = Config(
    region_name = 'ap-east-1',
    signature_version = 's3v4',
    s3 = {
        'addressing_style': 'virtual'
    }
)

client = boto3.client('s3', config=my_config)

url = client.generate_presigned_url(
ClientMethod='get_object', 
Params={'Bucket': 'sdkvjhwelkjf', 'Key': 'xxx.png'},
ExpiresIn=3600)

print(url)

signature_version = 's3v4',