search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
3.01k stars 249 forks source link

S3 SignatureDoesNotMatch #661

Closed DavidSouther closed 1 year ago

DavidSouther commented 1 year ago

Describe the bug

In this glue example, I am uploading a .py script for the crawler to run.

https://github.com/DavidSouther/aws-doc-sdk-examples/blob/4815350f9994b84e552d51672dd5dc544e53ee5c/rust_dev_preview/glue/src/bin/scenario.rs#L256-L266

2022-11-09T00:33:49.221508Z TRACE prepare_job:send_operation{operation="PutObject" service="s3"}: aws_sigv4::http_request::sign: canonical_request=PUT
/%2Frust-glue-mvp-53e51165-1cbb-47b1-bd10-d0bf8f726450-bucket/job.py
x-id=PutObject
content-length:3157
content-type:application/octet-stream
host:s3.us-east-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20221109T003349Z
x-amz-user-agent:aws-sdk-rust/0.51.0 api/s3/0.21.0 os/macos lang/rust/1.65.0

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent
UNSIGNED-PAYLOAD

...

2022-11-09T00:33:49.301059Z TRACE prepare_job:send_operation{operation="PutObject" service="s3"}:load_response: aws_smithy_http::middleware: http_response=Response { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "58QER2YAQD6GX0TH", "x-amz-id-2": "jzvNyWzH933f6CUgunszWXtmwx7xNftjy9NxWlMiesgCbjgYu19DwikK0ZPWEe6FzXHWEasCFOc=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Wed, 09 Nov 2022 00:33:48 GMT", "server": "AmazonS3", "connection": "close"}, body: b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAZTHI4CYE3PXHLYMJ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20221109T003349Z\n20221109/us-east-1/s3/aws4_request\nfcd623245a6d82d72cbcdcbb96516abcb255eb87f1535f63d04f89cf28c8929e</StringToSign><SignatureProvided>f1f7a993b438935d3ba91007aee28b37742e3b5e2e9c55e3df198351da54124b</SignatureProvided><StringToSignBytes>    [... 133 bytes ...]     </StringToSignBytes><CanonicalRequest>PUT\n//rust-glue-mvp-53e51165-1cbb-47b1-bd10-d0bf8f726450-bucket/job.py\nx-id=PutObject\ncontent-length:3157\ncontent-type:application/octet-stream\nhost:s3.us-east-1.amazonaws.com\nx-amz-content-sha256:UNSIGNED-PAYLOAD\nx-amz-date:20221109T003349Z\nx-amz-user-agent:aws-sdk-rust/0.51.0 api/s3/0.21.0 os/macos lang/rust/1.65.0\n\ncontent-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent\nUNSIGNED-PAYLOAD</CanonicalRequest><CanonicalRequestBytes>     [...418 bytes...]     </CanonicalRequestBytes><RequestId>58QER2YAQD6GX0TH</RequestId><HostId>jzvNyWzH933f6CUgunszWXtmwx7xNftjy9NxWlMiesgCbjgYu19DwikK0ZPWEe6FzXHWEasCFOc=</HostId></Error>" }

Expected Behavior

PutObject to succeed with no errors.

Current Behavior

PutObject is erroring with signature mismatches.

Reproduction Steps

src/main.rs (or src/bin/ssccee.rs)

use aws_config;
use aws_sdk_s3::types::ByteStream;

#[tokio::main]
pub async fn main() {
    let sdk_config = aws_config::load_from_env().await;
    let s3 = aws_sdk_s3::Client::new(&sdk_config);
    let bucket = format!("test-bucket-{}", uuid::Uuid::new_v4());
    let create_bucket = s3
        .create_bucket()
        .bucket(bucket)
        .send()
        .await
        .expect("create bucket");

    let bucket = create_bucket
        .location()
        .expect("Failed to get created bucket");

    s3.put_object()
        .bucket(bucket)
        .key("job.py")
        .body(
            ByteStream::from_path("resources/flight_etl_job_script.py".to_string())
                .await
                .expect("reading job python"),
        )
        .send()
        .await
        .expect("Uploaded object");
}
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

print("Hello, world")

Possible Solution

No response

Additional Information/Context

No response

Version

glue-code-examples v0.1.0 (/Users/dpsouth/devel/aws/aws-doc-sdk-examples/rust_dev_preview/glue)
├── aws-config v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   ├── aws-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-smithy-eventstream v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   │   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   │   ├── aws-smithy-http-tower v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   │   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-sdk-sso v0.21.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   ├── aws-endpoint v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-sig-auth v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-sigv4 v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   │   ├── aws-smithy-eventstream v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-smithy-eventstream v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   │   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-http-tower v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-json v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   └── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-sdk-sts v0.21.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   ├── aws-endpoint v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-sig-auth v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-http-tower v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-query v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-xml v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-http-tower v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-json v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
├── aws-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
├── aws-sdk-glue v0.21.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   ├── aws-endpoint v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-sig-auth v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-http-tower v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-json v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
├── aws-sdk-s3 v0.21.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   ├── aws-endpoint v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-sig-auth v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-sigv4 v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-async v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-checksums v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd)
│   │   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   │   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-eventstream v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-http-tower v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-smithy-xml v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
│   ├── aws-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
├── aws-smithy-client v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
├── aws-smithy-http v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)
├── aws-smithy-types v0.51.0 (https://github.com/awslabs/aws-sdk-rust?branch=main#557778dd) (*)

Environment details (OS name and version, etc.)

macOS Monterey Version 12.6.1 Apple M1 Pro

Logs

No response

rcoh commented 1 year ago

linked: https://github.com/awslabs/smithy-rs/issues/1308

ysaito1001 commented 1 year ago

For the code in Reproduction Steps above, it is passing a return value from CreateBucketOutput::location to PutObject::bucket that causes a panic, because the reassigned variable bucket now contains a forward /. If we comment out

    let bucket = create_bucket
        .location()
        .expect("Failed to get created bucket");

from the repro step, it no longer panics.

When a user already has a bucket name in the form of a string literal, i.e. let bucket = format!("test-bucket-{}", uuid::Uuid::new_v4()); in the repro step, it feels more natural that the user keeps using that literal throughout the code rather than going out of his/her way to obtain a return value out of CreateBucketOutput::location and using it instead.

That said, as the repro step demonstrates, users can certainly run into this issue, and when it happens, it is not so straightforward for them to figure out what they did wrong. From an ergonomics perspective, it's worth addressing.

DavidSouther commented 1 year ago

In retrospect this is (as typical) S3's fault. What even is the purpose of the Location response field in this API?

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_ResponseSyntax

jdisanti commented 1 year ago

I think the signature mismatch issue was resolved a while back, and we forgot to close this issue. Feel free to re-open if you disagree.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.