endgame
commented
9 months ago In which region did you create the bucket? Although S3 is a global service, buckets are attached to regions and you might be signing your request for the wrong region.
Open nitinprakash96 opened 9 months ago
endgame
commented
9 months ago In which region did you create the bucket? Although S3 is a global service, buckets are attached to regions and you might be signing your request for the wrong region.
nitinprakash96
commented
9 months ago @endgame us-east-1 (same as defaultRegion in the above snippet)
endgame
commented
9 months ago Weird. Here are some things I'd like to see to help debug this:
--debug truenitinprakash96
commented
9 months ago Request dump:
[Client Request] {
host = nitin-testing.s3.us-east-1.amazonaws.com:443
secure = True
method = POST
target = Nothing
timeout = ResponseTimeoutMicro 70000000
redirects = 0
path = /airtravel.csv
query = ?select-type=2&select=
headers = x-amz-content-sha256: REDACTED; x-amz-date: 20231118T040907Z; host: nitin-testing.s3.us-east-1.amazonaws.com; authorization: AWS4-HMAC-SHA256 Credential=REDACTED20231118/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
body = <?xml version="1.0" encoding="UTF-8"?><SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Expression>Select * from S3Object limit 5</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><CSV><FieldDelimiter>,</FieldDelimiter><FileHeaderInfo>USE</FileHeaderInfo></CSV></InputSerialization><OutputSerialization><CSV><FieldDelimiter>,</FieldDelimiter><RecordDelimiter>
</RecordDelimiter></CSV></OutputSerialization></SelectObjectContentRequest>
}
[Client Response] {
status = 403 Forbidden
headers = <REDACTED>
}Via AWS cli:
➜ ~ aws s3api select-object-content --debug --bucket nitin-testing --key "airtravel.csv" --expression "select * from s3object limit 5" --expression-type 'SQL' \
--input-serialization '{"CSV": {"FieldDelimiter": ",", "FileHeaderInfo": "USE"}, "CompressionType": "NONE"}' \
--output-serialization '{"CSV": {"FieldDelimiter": ",", "RecordDelimiter": "\n"}}' "output.csv" --profile nitin-dev --region us-east-1
If you're testing with open data, a description of how you set up and populated the bucket so I can try stuff locally.
Well, my bucket is a private bucket. All I did was uploaded a small csv file and tried running s3 select API from a haskell script.
One difference I note between the request made by amazonka and aws cli is that aws s3api has Content-Lenght in the headers whereas amazonka does not. Could it be that it is used for calculating the signature?
chreekat
commented
8 months ago I'm also currently investigating a SignatureDoesNotMatch error, except I'm using Cloudflare's R2 and the request is a PutObject. The same code and command works fine with AWS S3!
I'll add a bit more data when I can.
chreekat
commented
8 months ago My problem is probably something altogether different, so I opened a different issue: #975
Trying out the s3 select feature by amazon using
SelectObjectContentAPI by amazonka results inSignatureDoesNotMatchdespite correct credentials. The following is the only function that fails out of all that are listed inexamples/src/S3.hs.I'm not sure if I am doing something wrong here or there's something wrong with the API itself.
Here's a small function that can be plugged into
example/src/S3.hsmodule:which results in he following error:
Also, I'm using
AWS_PROFILEfor calling these APIs.