durch / rust-s3

Rust library for interfacing with S3 API compatible services
MIT License
498 stars 195 forks source link

AWS returns `InvalidArgument` when multipart object includes metadata #352

Closed jcrossley3 closed 8 months ago

jcrossley3 commented 1 year ago

Describe the bug User-defined metadata -- header names with the x-amz-meta- prefix -- should only be sent with the initiate_multipart_upload. Assuming Bucket::with_extra_headers is the correct way to set metadata on the object, and since the headers are passed to every request involved in a multipart upload, AWS errors when it sees the metadata on the first chunk sent after the initiate.

To Reproduce We're uploading very large files, which triggers the multipart upload within put_object_stream:

        let mut headers = http::HeaderMap::new();
        headers.insert("x-amz-meta-version", VERSION.into());
        let bucket = bucket.with_extra_headers(headers);
        bucket.put_object_stream(&mut rdr, path).await?

Expected behavior The multipart upload should succeed with or without metadata.

Environment

Additional context I was thinking it might be useful to introduce something like Bucket::put_object_stream_with_metadata. If you're open to that, I can work up a PR

UPDATE: after looking at the code, I thought a less intrusive approach would be just to ignore the metadata headers for any multipart-related commands other than initiate. Longer term, I think it might make more sense to replace the _with_content_type fn's with _with_metadata as content-type is considered metadata in the AWS api.