search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
2.91k stars 245 forks source link

Incorrect s3 metadata in Presigned URLs #1139

Closed taylor1791 closed 1 month ago

taylor1791 commented 1 month ago

Describe the bug

When I try to create a presigned URL for a put object operation with metadata, the metadata is not added to the object. Comparing the relevant parts of the generated presigned URL with another sdk implementation makes the reason evident.

Rust: X-Amz-SignedHeaders=host%3Bx-amz-meta-my-metadata JS: X-Amz-SignedHeaders=host&x-amz-meta-my-metadata=my-value

Notice that the Rust URL appears to append the x-amz-meta-my-metadata param to the X-Amz-SignedHeaders=host param separated by an encoded semicolon? It also abstains from including the value.

Entire Rust URL: https://my-bucket.s3.us-east-1.amazonaws.com/my-key?x-id=PutObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ANOTREAL%2F20240508%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240508T044525Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host%3Bx-amz-meta-my-metadata&X-Amz-Signature=ee22808d868692b2254b1ce4b66f5d5f075d1b179132c14e54b523cb7cacb250 Entire JS URL: https://my-bucket.s3.us-east-1.amazonaws.com/my-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=dummy%2F20240508%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240508T045407Z&X-Amz-Expires=3600&X-Amz-Signature=0241d68acc6847303e858bd8fd53ac306f726eda933bfe71c8a1efaaa920f146&X-Amz-SignedHeaders=host&x-amz-meta-my-metadata=my-value&x-id=PutObject

Expected Behavior

I expected the presigned method to generate a URL that properly includes the specified metadata.

Current Behavior

The metadata is improperly append to the X-Amz-SignedHeaders param.

Reproduction Steps

#[tokio::main]
async fn main() {
    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .test_credentials()
        .region("us-east-1")
        .load()
        .await;

    let s3 = aws_sdk_s3::Client::new(&config);

    let presigned_config = aws_sdk_s3::presigning::PresigningConfig::builder()
        .expires_in(std::time::Duration::from_secs(3600))
        .build()
        .unwrap();

    let presigned_request = s3.put_object()
        .bucket("my-bucket")
        .key("my-key")
        .metadata("my-metadata", "my-value")
        .presigned(presigned_config)
        .await
        .unwrap();

    println!("Presigned URL: {}", presigned_request.uri());
}
const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");

async function main() {
  const client = new S3Client({
    region: "us-east-1",
  });

  const command = new PutObjectCommand({
    Bucket: "my-bucket",
    Key: "my-key",
    Metadata: {
      "filename": "file.txt",
    },
  });

  const url = await getSignedUrl(client, command, { expiresIn: 3600 })

  console.log(url);
}

main();

Possible Solution

No response

Additional Information/Context

No response

Version

Version ```text ├── aws-config v1.3.0 │ ├── aws-credential-types v1.2.0 │ │ ├── aws-smithy-async v1.2.1 │ │ ├── aws-smithy-runtime-api v1.6.0 │ │ │ ├── aws-smithy-async v1.2.1 (*) │ │ │ ├── aws-smithy-types v1.1.9 │ │ ├── aws-smithy-types v1.1.9 (*) │ ├── aws-runtime v1.2.1 │ │ ├── aws-credential-types v1.2.0 (*) │ │ ├── aws-sigv4 v1.2.1 │ │ │ ├── aws-credential-types v1.2.0 (*) │ │ │ ├── aws-smithy-eventstream v0.60.4 │ │ │ │ ├── aws-smithy-types v1.1.9 (*) │ │ │ ├── aws-smithy-http v0.60.8 │ │ │ │ ├── aws-smithy-eventstream v0.60.4 (*) │ │ │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ │ │ ├── aws-smithy-types v1.1.9 (*) │ │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-smithy-async v1.2.1 (*) │ │ ├── aws-smithy-eventstream v0.60.4 (*) │ │ ├── aws-smithy-http v0.60.8 (*) │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-types v1.2.0 │ │ │ ├── aws-credential-types v1.2.0 (*) │ │ │ ├── aws-smithy-async v1.2.1 (*) │ │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ │ ├── aws-smithy-types v1.1.9 (*) │ ├── aws-sdk-sso v1.22.0 │ │ ├── aws-credential-types v1.2.0 (*) │ │ ├── aws-runtime v1.2.1 (*) │ │ ├── aws-smithy-async v1.2.1 (*) │ │ ├── aws-smithy-http v0.60.8 (*) │ │ ├── aws-smithy-json v0.60.7 │ │ │ └── aws-smithy-types v1.1.9 (*) │ │ ├── aws-smithy-runtime v1.4.0 │ │ │ ├── aws-smithy-async v1.2.1 (*) │ │ │ ├── aws-smithy-http v0.60.8 (*) │ │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-types v1.2.0 (*) │ ├── aws-sdk-ssooidc v1.22.0 │ │ ├── aws-credential-types v1.2.0 (*) │ │ ├── aws-runtime v1.2.1 (*) │ │ ├── aws-smithy-async v1.2.1 (*) │ │ ├── aws-smithy-http v0.60.8 (*) │ │ ├── aws-smithy-json v0.60.7 (*) │ │ ├── aws-smithy-runtime v1.4.0 (*) │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-types v1.2.0 (*) │ ├── aws-sdk-sts v1.22.0 │ │ ├── aws-credential-types v1.2.0 (*) │ │ ├── aws-runtime v1.2.1 (*) │ │ ├── aws-smithy-async v1.2.1 (*) │ │ ├── aws-smithy-http v0.60.8 (*) │ │ ├── aws-smithy-json v0.60.7 (*) │ │ ├── aws-smithy-query v0.60.7 │ │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-smithy-runtime v1.4.0 (*) │ │ ├── aws-smithy-runtime-api v1.6.0 (*) │ │ ├── aws-smithy-types v1.1.9 (*) │ │ ├── aws-smithy-xml v0.60.8 │ │ ├── aws-types v1.2.0 (*) │ ├── aws-smithy-async v1.2.1 (*) │ ├── aws-smithy-http v0.60.8 (*) │ ├── aws-smithy-json v0.60.7 (*) │ ├── aws-smithy-runtime v1.4.0 (*) │ ├── aws-smithy-runtime-api v1.6.0 (*) │ ├── aws-smithy-types v1.1.9 (*) │ ├── aws-types v1.2.0 (*) ├── aws-sdk-s3 v1.26.0 │ ├── aws-credential-types v1.2.0 (*) │ ├── aws-runtime v1.2.1 (*) │ ├── aws-sigv4 v1.2.1 (*) │ ├── aws-smithy-async v1.2.1 (*) │ ├── aws-smithy-checksums v0.60.7 │ │ ├── aws-smithy-http v0.60.8 (*) │ │ ├── aws-smithy-types v1.1.9 (*) │ ├── aws-smithy-eventstream v0.60.4 (*) │ ├── aws-smithy-http v0.60.8 (*) │ ├── aws-smithy-json v0.60.7 (*) │ ├── aws-smithy-runtime v1.4.0 (*) │ ├── aws-smithy-runtime-api v1.6.0 (*) │ ├── aws-smithy-types v1.1.9 (*) │ ├── aws-smithy-xml v0.60.8 (*) │ ├── aws-types v1.2.0 (*) ```

Environment details (OS name and version, etc.)

Linux 6.1.67

Logs

Logs 2024-05-08T05:08:20.701471Z DEBUG aws_sdk_s3::endpoint_lib: loading default partitions 2024-05-08T05:08:20.721860Z DEBUG hyper_rustls::config: with_native_roots processed 146 valid and 0 invalid certs 2024-05-08T05:08:20.722150Z DEBUG invoke{service=s3 operation=PutObject}:apply_configuration: aws_smithy_runtime::client::orchestrator: timeout settings for this operation: TimeoutConfig { connect_timeout: Set(3.1s), read_timeout: Disabled, operation_timeout: Disabled, operation_attempt_timeout: Disabled } 2024-05-08T05:08:20.722305Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase 2024-05-08T05:08:20.722442Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase 2024-05-08T05:08:20.722515Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request 2024-05-08T05:08:20.722549Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1 2024-05-08T05:08:20.722626Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: resolving endpoint endpoint_params=EndpointResolverParams(TypeErasedBox[!Clone]:Params { bucket: Some("my-bucket"), region: Some("us-east-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("my-key"), prefix: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }) endpoint_prefix=None 2024-05-08T05:08:20.722981Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: will use endpoint Endpoint { url: "https://my-bucket.s3.us-east-1.amazonaws.com", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingName": String("s3"), "signingRegion": String("us-east-1"), "disableDoubleEncoding": Bool(true)})])} } 2024-05-08T05:08:20.723173Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt:lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took 0ns) new_expiration=2024-05-08T05:23:20.721919Z valid_for=900s partition=IdentityCachePartition(0) 2024-05-08T05:08:20.723236Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime::client::identity::cache::lazy: loaded identity 2024-05-08T05:08:20.723678Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator: ending orchestration early because the stop point is `BeforeTransmit` 2024-05-08T05:08:20.723778Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
aajtodd commented 1 month ago

Thanks for the issue.

It looks like it is being added but not in the way you expect. A presigned request returned by the SDK also includes things like the method and HTTP headers to send. Dumping a debug version of the request shows the metadata is added as a header:

    println!("presigned req: {:#?}", presigned_request);

Output:

presigned req: PresignedRequest {
    method: "PUT",
    uri: "https://my-bucket.s3.us-east-1.amazonaws.com/my-key?x-id=PutObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ANOTREAL%2F20240508%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240508T125034Z&X-Amz-Expir
es=3600&X-Amz-SignedHeaders=host%3Bx-amz-meta-my-metadata&X-Amz-Signature=9ef755e71e52706b76c923040627056ad84d63ed358827cedb6a9fd1fa4832e6",
    headers: Headers {
        headers: {
            "x-amz-meta-my-metadata": HeaderValue {
                _private: H0(
                    "my-value",
                ),
            },
        },
    },
}

Whereas JS is hoisting all the headers it can into query parameters.

If you convert the request directly into an HTTP request and send it with the headers it should work. Alternatively you can do something similar as JS is doing and hoist the returned headers and construct a new URI with them present as a workaround.

I've opened https://github.com/awslabs/aws-sdk-rust/issues/1140 to track potentially improving/adding this at some point.

taylor1791 commented 1 month ago

Thanks for politely correcting my misunderstanding. Indeed it does work if I send the headers with the request. I've given the new issue a thumbs up.

github-actions[bot] commented 1 month ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.