trinodb / aws-proxy

Proxy for S3
Apache License 2.0
7 stars 3 forks source link

Support multiple Content-Encodings #100

Closed vagaerg closed 1 month ago

vagaerg commented 2 months ago

In the context of chunked uploads we've observed that callers can legally provide content-encoding: aws-chunked, <something-else> as described in the docs

This extra metadata should become object metadata on S3, meaning S3 would use the aws-chunked value to understand the format of the incoming object, but store only the <something-else> part as metadata for the object.

Since the proxy uploads data to S3 in a single chunk (i.e., we never pass down aws-chunked data), we should make sure we still respect the content-type provided by the caller.

Also note that requests may have multiple content-encoding values and we should make sure the proxy can understand this regardless