search

Netflix / Priam

Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
Apache License 2.0
1.04k stars 294 forks source link

Encrypted S3 buckets #655

Open zmarois opened 6 years ago

zmarois commented 6 years ago

While I know that Priam has data encryption capabilities, I'd like to utilize S3 default encryption to save myself the hassle of managing encryption keys on my servers.

However, when I turn on default encryption, the etags in the response no longer match the request (presumably because they are the md5 of the encrypted data). The multipart part uploader validates the etag on each part, despite also sending it to S3 for Amazon to validate, so the uploads of large files fails. Since I didn't see the value in the extra (and now invalid) check, I just commented it out in my fork.

Am I missing something in the value in the extra check? Is this worth pushing back into upstream (As a code deletion instead)? How about having the extra check configurable to run?

arunagrawal84 commented 6 years ago

We can make this check configurable based on if the s3 bucket is using the default encryption. I believe that by using getBucketEncryption() method this check could be skipped at runtime if the s3 bucket is using default encryption type. I have personally not seen this check ever getting triggered in our fleet but I don't see a value of removing either if you are not using default encryption. Making it configurable though could be a happy medium.

zmarois commented 6 years ago

Awesome. I might push this back up stream based on the bucket encryption. Thanks.