search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.94k stars 1.86k forks source link

Add Client Side Encryption to S3 client #3395

Open frittentheke opened 2 years ago

frittentheke commented 2 years ago

Describe the feature

While there is support for client-side encryption in other popular language SDKs like Java, Go or C++ (https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html) for encrypting data before it's sent to AWS S3 this is not supported by boto3 (Python3).

The documentation does talk about using the Encryption SDK (https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingClientSideEncryption.html), but this is produces a different data format:

The AWS Encryption SDK and the Amazon S3 encryption clients are not compatible because they produce ciphertexts with different data formats. For more information about the AWS Encryption SDK, see the AWS Encryption SDK Developer Guide.

(https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html)

Use Case

With no support for the otherwise common format for client side encryption, that is used for other language SDKs, it's not possible to e.g. write encrypted objects via Java and to read them back via a Python3 client using boto3.

Also there are quite a few tools using boto3 as S3 client library which would then allow for client side encryption to be used:

Proposed Solution

There have been attempts to wrap the boto3 client and to add client encryption externally, like

  1. @boldfield implemented via https://github.com/boldfield/s3-encryption, continued by @stephensorriaux (https://github.com/StephenSorriaux/s3-encryption)
  2. @terrycain added this for aioboto3, see https://github.com/terrycain/aioboto3/blob/07d6cea1214a9bb6529f334e375134554ad7ab58/README.rst#amazon-s3-client-side-encryption, unfortunately only for KMS managed keys yet
  3. ....

Proposed solution would be to have this feature natively provided by boto3 and to make the SDKs compatible in this regard.

Other Information

No response

Acknowledgements

SDK version used

any

Environment details (OS name and version, etc.)

Linux

StephenSorriaux commented 2 years ago

Hey,

Thanks for citing my work here. FWIW my fork also has an (undocumented) async client available, and I was able use the Java SDK to decrypt what my Python lib encrypted (which was awesome to see).

tim-finnigan commented 2 years ago

Thanks @frittentheke for the feature request. The cryptography team is working towards taking ownership of the S3 Encryption Client implementations in various languages. Supporting boto3 is on their roadmap but they don’t have a concrete timeline as of yet.

frittentheke commented 2 years ago

Thanks @frittentheke for the feature request. The cryptography team is working towards taking ownership of the S3 Encryption Client implementations in various languages. Supporting boto3 is on their roadmap but they don’t have a concrete timeline as of yet.

Thanks for the quick and positive response! Is the roadmap accessible anywhere or is there another issue that could be tracked for this feature?

tim-finnigan commented 2 years ago

Hi @frittentheke thanks for following up. There is no public roadmap for this to my knowledge. But the Cryptography team said this feature isn't likely to be implemented in the immediate future. I recommend reaching out through AWS Support to seek more information as needed.

frittentheke commented 1 year ago

Hi @frittentheke thanks for following up. There is no public roadmap for this to my knowledge. But the Cryptography team said this feature isn't likely to be implemented in the immediate future. I recommend reaching out through AWS Support to seek more information as needed.

How would support be able to help here @tim-finnigan? Could you kindly mention anybody from that particular team to maybe get a feel for how far up in the queue the client-side-encryption feature is?

tim-finnigan commented 1 year ago

Hi @frittentheke - I suggested reaching out to AWS Support as that could help with establishing a more direct correspondence regarding this particular feature request going forward. The information passed on to me by the cryptography team was that this is on their roadmap but they don't yet have an official timeline, and as I mentioned this isn't something that is likely to prioritized in the near future. But if others share their use cases and how the lack of this feature is a blocker for them then it could help with escalating the priority.

timetxt commented 1 year ago

tim-finnigan

AWS Support has the same or actually less visibility and connection with cryptography team comparing with anyone this ticket. Reaching out AWS Support will not give the information that you are looking for.

timetxt commented 1 year ago

also for anyone found this issue post, s3crypto is supported in language like Go. AWS SDK for Go has two versions, V1 and V2. V1 is still supporting s3crypto while s3crypto is removed from V2 at the moment

V1: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3crypto/#pkg-overview

V2 document when s3crypto was supported in V2: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2@v0.20.0/service/s3/s3crypto

ristosaari commented 3 months ago

I would appreciate if Python boto3 support would be given out of the box fox SDK.

medmunds commented 4 days ago

Another use case: Amazon SES's email receiving has a "message encryption" option for its "Deliver to S3 bucket" action. But it requires using an S3 encryption client to decrypt the S3 objects: https://docs.aws.amazon.com/ses/latest/dg/receiving-email-action-s3.html.

There is currently no AWS-supported Python option for using SES email receiving with S3 message encryption enabled. (And the third-party Python packages seem to be outdated.)

briveira commented 2 days ago

Two years later, AWS Python SDK still lacks client side encryption with S3. Can't believe it! :-o