search

durch / rust-s3

Rust library for interfacing with S3 API compatible services
MIT License
498 stars 195 forks source link

Rust-s3 can send a Authorization and X-Amz-Date header with mismatching dates #308

Open ahicks92 opened 1 year ago

ahicks92 commented 1 year ago

Describe the bug A clear and concise description of what the bug is.

I have a CLI program to upload keys to Minio that's testing a bunch of logic that's unfortunately not standalone, so I'll get it standalone if we need it. I call new_with_path_style for minio, and create credentials with an access/secret key directly. We get:

localhost:9000 [REQUEST s3.PutObject] [2022-10-24T13:23:57.333] [Client IP: 127.0.0.1]
localhost:9000 PUT /test/testing
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 Content-Length: 90
localhost:9000 Content-Md5: 0mxWL5x5zssg9ukexLvGQQ==
localhost:9000 Content-Type: application/octet-stream
localhost:9000 X-Amz-Content-Sha256: 2bbfbfc647f103476230324d0496560267ded04b1d1b89446d5f79733ba70fa3
localhost:9000 Accept: */*
localhost:9000 Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20221024/http://localhost:9000/s3/aws4_request,Sign
edHeaders=content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date,Signature=d6e73c62a92aca6724cd2e5
c312a691897b473e7f47493530e8a265204f567c5
localhost:9000 X-Amz-Date: 20221024T202357Z
localhost:9000 Date: Mon, 24 Oct 2022 20:23:57 +0000
localhost:9000 <BODY>
localhost:9000 [RESPONSE] [2022-10-24T13:23:57.336] [ Duration 3.25ms  ↑ 103 B  ↓ 430 B ]
localhost:9000 400 Bad Request
localhost:9000 Content-Type: application/xml
localhost:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
localhost:9000 Vary: Origin,Accept-Encoding
localhost:9000 X-Amz-Request-Id: 17211B031FA921FD
localhost:9000 X-Content-Type-Options: nosniff
localhost:9000 X-Xss-Protection: 1; mode=block
localhost:9000 Accept-Ranges: bytes
localhost:9000 Content-Length: 430
localhost:9000 Content-Security-Policy: block-all-mixed-content
localhost:9000 Server: MinIO
localhost:9000 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; incorrect da
te format. This date in the credential must be in the format &#34;yyyyMMdd&#34;.</Message><Key>testing</Key><BucketName>
test</BucketName><Resource>/test/testing</Resource><RequestId>17211B031FA921FD</RequestId><HostId>df21c7f3-f2b5-4129-b1b
9-0b7e75b13991</HostId></Error>
localhost:9000

At first I thought it was a bug with Minio, but if we look at AWS troubleshooting: https://docs.aws.amazon.com/general/latest/gr/signature-v4-troubleshooting.html

They say that we get a very similar error message if both the Authorization and X-Amz-Date headers mismatch (so, to be honest, I'm not sure how this is working on AWS either? But maybe it sends different parameters there, I'm not sure yet).

To Reproduce Steps to reproduce the behavior (ideally with a minimal code sample):

See the description. I'll get a stripped down code sample if needed.

Expected behavior A clear and concise description of what you expected to happen.

This doesn't error.

Environment

ahicks92 commented 1 year ago

Sorry, edited for a few typos and such. Not sure how I mismatched Credential and Authorization in my head while flipping windows. My bad.

ahicks92 commented 1 year ago

here's a minimal repro program:

use s3::bucket::Bucket;
use s3::creds::Credentials;

#[tokio::main(flavor = "multi_thread")]
async fn main() {
    let creds = Credentials::new(Some("minioadmin"), Some("minioadmin"), None, None, None).unwrap();
    let bucket = Bucket::new_with_path_style("test", "http://localhost:9000".parse().unwrap(), creds).unwrap();

    bucket
        .put_object("testing2", "asdfasdf".as_bytes())
        .await
        .unwrap();
}

And minio is launched like this:

#!/bin/bash
docker run --network=host \
  -v ~/minio:/data \
  -e MINIO_HTTP_TRACE=/data/trace.txt \
  -e MINIO_DOMAIN=localhost \
  quay.io/minio/minio \
  server /data --console-address="0.0.0.0:9001"

When I run it:

    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `/home/ahicks/repos/backtrace-rust/target/debug/upload_key`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Http(400, "<?xml version=\"1.0\" encoding=\"UTF-
8\"?>\n<Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; incor
rect date format. This date in the credential must be in the format &#34;yyyyMMdd&#34;.</Message><Key>testing2</Key><Buc
ketName>test</BucketName><Resource>/test/testing2</Resource><RequestId>172120BA37695E7C</RequestId><HostId>df21c7f3-f2b5
-4129-b1b9-0b7e75b13991</HostId></Error>")', s3_sink/src/bin/upload_key.rs:12:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

From the minio mc admin trace:

ahicks@ahicks-work-vm:~/repos$ mc admin trace --all --verbose lh
localhost:9000 [REQUEST s3.PutObject] [2022-10-24T15:09:11.997] [Client IP: 127.0.0.1]
localhost:9000 PUT /test/testing2
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 Date: Mon, 24 Oct 2022 22:09:11 +0000
localhost:9000 X-Amz-Content-Sha256: 2413fb3709b05939f04cf2e92f7d0897fc2596f9ad0b8a9ea855c7bfebaae892
localhost:9000 Content-Md5: aiBL2J88g0iv1cd8cXoJeg==
localhost:9000 Content-Type: application/octet-stream
localhost:9000 Content-Length: 8
localhost:9000 X-Amz-Date: 20221024T220911Z
localhost:9000 Accept: */*
localhost:9000 Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20221024/http://localhost:9000/s3/aws4_request,Sign
edHeaders=content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date,Signature=86c9e0b653a9ffdca6bc26e
c2fd31ea74f0065a457d68b7fa5667a9b02265e59
localhost:9000 <BODY>
localhost:9000 [RESPONSE] [2022-10-24T15:09:11.997] [ Duration 204µs  ↑ 103 B  ↓ 432 B ]
localhost:9000 400 Bad Request
localhost:9000 Server: MinIO
localhost:9000 X-Xss-Protection: 1; mode=block
localhost:9000 Content-Security-Policy: block-all-mixed-content
localhost:9000 Content-Length: 432
localhost:9000 Content-Type: application/xml
localhost:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
localhost:9000 Vary: Origin,Accept-Encoding
localhost:9000 X-Amz-Request-Id: 172120C15EF42E63
localhost:9000 X-Content-Type-Options: nosniff
localhost:9000 Accept-Ranges: bytes
localhost:9000 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; incorrect da
te format. This date in the credential must be in the format &#34;yyyyMMdd&#34;.</Message><Key>testing2</Key><BucketName
>test</BucketName><Resource>/test/testing2</Resource><RequestId>172120C15EF42E63</RequestId><HostId>df21c7f3-f2b5-4129-b
1b9-0b7e75b13991</HostId></Error>
localhost:9000
ahicks@ahicks-work-vm:~/repos$

If I mc cp a file, we can see what it's sending, roughly:

mc: <DEBUG> PUT /test/testing HTTP/1.1
Host: localhost:9000
User-Agent: MinIO (linux; amd64) minio-go/v7.0.41 mc/RELEASE.2022-10-20T23-26-33Z
Content-Length: 263
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20221024/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-conten
t-sha256;x-amz-date;x-amz-decoded-content-length,Signature=**REDACTED**
Content-Type: application/rls-services+xml
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20221024T221102Z
X-Amz-Decoded-Content-Length: 90
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 24 Oct 2022 22:11:02 GMT
Etag: "d26c562f9c79cecb20f6e91ec4bbc641"
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 172120DB344EB7E2
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  2.024914ms

...3_sink/src/lib.rs: 90 B / 90 B ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.25 KiB/s 0s

And from the Minio trace for that:

localhost:9000 [REQUEST s3.PutObject] [2022-10-24T15:11:02.951] [Client IP: 127.0.0.1]
localhost:9000 PUT /test/testing
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 X-Amz-Decoded-Content-Length: 90
localhost:9000 Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20221024/us-east-1/s3/aws4_request,SignedHeaders=ho
st;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=c10e7aa2b2a78d9b4eb2795c7958d5b48c9d9edf4be452
a6b3cb7d982dd2cf03
localhost:9000 Content-Length: 263
localhost:9000 Content-Type: application/rls-services+xml
localhost:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.41 mc/RELEASE.2022-10-20T23-26-33Z
localhost:9000 X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
localhost:9000 X-Amz-Date: 20221024T221102Z
localhost:9000 <BODY>
localhost:9000 [RESPONSE] [2022-10-24T15:11:02.952] [ Duration 1.407ms  ↑ 382 B  ↓ 0 B ]
localhost:9000 200 OK
localhost:9000 Content-Security-Policy: block-all-mixed-content
localhost:9000 ETag: "d26c562f9c79cecb20f6e91ec4bbc641"
localhost:9000 Vary: Origin,Accept-Encoding
localhost:9000 X-Amz-Request-Id: 172120DB344EB7E2
localhost:9000 X-Content-Type-Options: nosniff
localhost:9000 Accept-Ranges: bytes
localhost:9000 Content-Length: 0
localhost:9000 X-Xss-Protection: 1; mode=block
localhost:9000 Server: MinIO
localhost:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
localhost:9000 <BODY>
localhost:9000
durch commented 1 year ago

@ahicks92 thanks for an in depth report, I'll find some time to look into this

puuuuh commented 1 year ago

Thats because region name contains slashes, you need to create Region::Custom with the same endpoint, but different name

puuuuh commented 1 year ago

And probably its better to use some hardcoded name for FromStr impl

ahicks92 commented 1 year ago

Good to know, but sadly we have left rust-s3 for other options (not because of this). So i'm no longer in a position to easily verify. Do we want to leave this open? If nothing else docs should maybe be made clearer.

chris13524 commented 1 year ago

According to the Rustdoc on the region type, "Custom region requires valid region name and endpoint". Why is this?

Perhaps in some S3 implementations, a non-arbitrary region name would be important? In my case, it doesn't seem like Minio cares what the region name is.