Open nitinprakash96 opened 9 months ago
In which region did you create the bucket? Although S3 is a global service, buckets are attached to regions and you might be signing your request for the wrong region.
@endgame us-east-1
(same as defaultRegion
in the above snippet)
Weird. Here are some things I'd like to see to help debug this:
--debug true
Request dump:
[Client Request] {
host = nitin-testing.s3.us-east-1.amazonaws.com:443
secure = True
method = POST
target = Nothing
timeout = ResponseTimeoutMicro 70000000
redirects = 0
path = /airtravel.csv
query = ?select-type=2&select=
headers = x-amz-content-sha256: REDACTED; x-amz-date: 20231118T040907Z; host: nitin-testing.s3.us-east-1.amazonaws.com; authorization: AWS4-HMAC-SHA256 Credential=REDACTED20231118/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
body = <?xml version="1.0" encoding="UTF-8"?><SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Expression>Select * from S3Object limit 5</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><CSV><FieldDelimiter>,</FieldDelimiter><FileHeaderInfo>USE</FileHeaderInfo></CSV></InputSerialization><OutputSerialization><CSV><FieldDelimiter>,</FieldDelimiter><RecordDelimiter>
</RecordDelimiter></CSV></OutputSerialization></SelectObjectContentRequest>
}
[Client Response] {
status = 403 Forbidden
headers = <REDACTED>
}
Via AWS cli:
➜ ~ aws s3api select-object-content --debug --bucket nitin-testing --key "airtravel.csv" --expression "select * from s3object limit 5" --expression-type 'SQL' \
--input-serialization '{"CSV": {"FieldDelimiter": ",", "FileHeaderInfo": "USE"}, "CompressionType": "NONE"}' \
--output-serialization '{"CSV": {"FieldDelimiter": ",", "RecordDelimiter": "\n"}}' "output.csv" --profile nitin-dev --region us-east-1
If you're testing with open data, a description of how you set up and populated the bucket so I can try stuff locally.
Well, my bucket is a private bucket. All I did was uploaded a small csv file and tried running s3 select API from a haskell script.
One difference I note between the request made by amazonka and aws cli is that aws s3api
has Content-Lenght
in the headers whereas amazonka does not. Could it be that it is used for calculating the signature?
I'm also currently investigating a SignatureDoesNotMatch
error, except I'm using Cloudflare's R2 and the request is a PutObject. The same code and command works fine with AWS S3!
I'll add a bit more data when I can.
My problem is probably something altogether different, so I opened a different issue: #975
Trying out the s3 select feature by amazon using
SelectObjectContent
API by amazonka results inSignatureDoesNotMatch
despite correct credentials. The following is the only function that fails out of all that are listed inexamples/src/S3.hs
.I'm not sure if I am doing something wrong here or there's something wrong with the API itself.
Here's a small function that can be plugged into
example/src/S3.hs
module:which results in he following error:
Also, I'm using
AWS_PROFILE
for calling these APIs.