search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.46k stars 729 forks source link

object-store: support for client-side encryption on S3 #4753

Open andrusha opened 1 year ago

andrusha commented 1 year ago

On S3 when objects are client-side encrypted the encryption metadata is stored and transmitted through headers on GET and HEAD requests, which is required to decrypt objects on the client-side.

Currently the ObjectMeta does not support provider-specific metadata although it's already able to extract it from request headers.

Describe the solution you'd like The minimal change would be to passthrough S3-specific client-side encryption metadata through the ObjectMeta, which could then by used by client.

Describe alternatives you've considered The better solution would be to support content decryption within the library itself, however it would create dependencies on openssl or similar and add considerable amount of logic, which might limit the types of clients who would be able to use this library.

Additional context I'm willing to author the change, but need guidance on the design.

tustvold commented 1 year ago

I've filed https://github.com/apache/arrow-rs/issues/4754 to track designing how best to support custom user metadata.

_FYI ring, which this crate already uses for request signing and TLS, also supports AES-GCM if you're looking to avoid an additional dependency on openssl_

tveldhui commented 2 months ago

Has this been resolved by #5915 ? thanks!

tustvold commented 2 months ago

I believe this feature requires additional headers that aren't yet supported. Although we are in a better position to support this than we were when this issue was originally filed, I do not know of anyone currently working on adding support for this

tveldhui commented 2 months ago

Thanks @tustvold, much appreciated. My basic understanding is that we'll need x-amz-meta-x-amz-key, x-amz-meta-x-amz-iv and x-amz-meta-x-amz-matdesc which seems to be covered by #5915 (it adds support for x-amz-meta user-defined headers?) Does that seem like enough, or is further work on object-store needed?