search

ceph / s3-tests

Compatibility tests for S3 clones
MIT License
296 stars 288 forks source link

Confusion with Date format and V4 signing in s3tests_boto3 test_object_create_date_and_amz_date #346

Open jamshid opened 4 years ago

jamshid commented 4 years ago

Hi, I'm very confused by the test s3tests_boto3.functional.test_headers:test_object_create_date_and_amz_date https://github.com/ceph/s3-tests/blob/master/s3tests_boto3/functional/test_headers.py#L344

1) Test seems to be attempting to send both the Date and X-Amz-Date header, but only the Date header is actually getting sent. These are the versions from bootstrap, in case that matters:

virtualenv/lib/python2.7/site-packages/boto-2.49.0.dist-info virtualenv/lib/python2.7/site-packages/boto3-1.13.2.dist-info virtualenv/lib/python2.7/site-packages/botocore-1.16.2.dist-info

2) It fails against AWS S3 because the AWS signs with the Date header as-is. I thought V4 sigs require converting the Date value to ISO 8601 (without dashes or colons). The test fails with this 403 from AWS:

The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJV3VAEW6OFKXWMLQ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
Wed, 06 May 2020 15:42:30 -0000
20200506/us-east-1/s3/aws4_request

The test computes the StringToSign with the ISO8601 formatting of Date. Note the line after "AWS4-HMAC-SHA256".

botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
PUT
/jam-ceph-a1/foo

date:Wed, 06 May 2020 15:42:30 -0000
host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

date;host;x-amz-content-sha256
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20200506T154230Z
20200506/us-east-1/s3/aws4_request
dc8caa951c884d6000554256c19076987fa07d3119c5d06a49834977dcf2179f

3) I don't know how AWS can be "wrong" here but this from the jets3t library implies the Date header should be converted to ISO 8601 in the signature, right? Maye there was a change or there are differences in regions? https://jets3t.s3.amazonaws.com/api/org/jets3t/service/utils/SignatureUtils.html#awsV4ParseAndFormatDate-org.apache.http.client.methods.HttpUriRequest-

jamshid commented 4 years ago

This is from Amazon S3 Glacier but fwiw it says Date must be ISO 8601. So I guess the test should expect failure when it sends Date like an HTTP date, without a x-amz-date header? https://docs.aws.amazon.com/amazonglacier/latest/dev/api-common-request-headers.html