KMSEncryptionMaterialsProvider equivalent in boto3

[owenrumney]

I've been looking at the Java SDK for client side encryption up to AWS using KMS keys and have used KMSEncryptionMaterialsProvider to do so as per this blog http://java.awsblog.com/post/Tx19OLB7M3D6DS8/S3-Encryption-with-AWS-Key-Management-Service

Is there an equivalent for boto3 to client side encrypt or is it a case of generating a key, encrypting with that key then pushing up to S3?

Thanks, Owen

[danielgtaylor]

At the moment there is no such functionality in Boto 3, however future hand-written customizations could provide something similar.

[jAlpedrinha]

Hi, I've been facing somehow the same issue, and I found a lib that provides compatibility with the Ruby SDK: https://github.com/boldfield/s3-encryption

I've not been able to test thoroughly but I believe it could help. I'm on a project that requires client side encryption but is unfortunately built around the original boto. I'll subscribe this issue and if I come to use boto3 for this purpose I'd be glad to help.

[ty-dev]

Seems like this should be pretty doable. Started working on it for client side with AWS KMS, but ran into some issues with decrypting the envelope key. Will update once it is working. Here is current code snippet:

import base64 from Crypto.Cipher import AES

encrypted = bucket.get_key(object_key) metadata = encrypted.metadata envelope_key = base64.b64decode(metadata['x-amz-key-v2']) envelope_iv = base64.b64decode(metadata['x-amz-iv'])

encryption_key = kms.decrypt(CiphertextBlob=envelope_key)

[ty-dev]

Got it working with KMS CMK key. Works with client side sdk for ruby and java. Used the following link as a starting point: http://stackoverflow.com/questions/29784535/how-to-decrypt-aws-ruby-client-side-encryption-in-python

Same code as previous comment for decrypting envelope key with one tweak:

import base64
import json
from Crypto.Cipher import AES

encrypted = bucket.get_key(object_key)
metadata = encrypted.metadata
envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encrypt_ctx = json.loads(metadata['x-amz-matdesc'])

encryption_key = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)

[tedder]

I'm trying to do the kms.decrypt thing and failing. @ty-dev does the above work for you?

http://stackoverflow.com/questions/34957677/invalidciphertextexception-when-calling-kms-decrypt-with-s3-metadata

[ty-dev]

Hey Tedder,

Here is the code I got working to decrypt Java KMS client side uploads. Its hacky but was enough to prove out the solution. I did the encryption side for both python and dotnet as well if needed.

from __future__ import print_function
import boto
import boto3
import tempfile
import base64
import json
import Crypto
from Crypto.Cipher import AES
from Crypto import Random
import os, random, struct

# decrypt_file method from: http://eli.thegreenplace.net/2010/06/25/aes-encryption-of-files-in-python-with-pycrypto
def decrypt_file(key, in_filename, iv, original_size, out_filename=None, chunksize=16*1024):

    if not out_filename:
        out_filename = 'tempfile.png'

    with open(in_filename, 'rb') as infile:

        decryptor = AES.new(key, AES.MODE_CBC, iv)

        with open(out_filename, 'wb') as outfile:
            while True:
                chunk = infile.read(chunksize)
                if len(chunk) == 0:
                    break
                outfile.write(decryptor.decrypt(chunk))

            outfile.truncate(original_size)

REGION = '***'
BUCKET = '***'
s3_key = '***.png'
filename = '***.png'

s3 = boto3.client('s3')
kms = boto3.client('kms')

# download encrypted object from S3
encrypted = s3.get_object(Bucket=BUCKET,Key=s3_key)

# get object metadata from encrypted object and decode base64 strings
metadata = encrypted['Metadata']
envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encrypt_ctx = json.loads(metadata['x-amz-matdesc'])
original_size = metadata['x-amz-unencrypted-content-length']

# use AWS KMS to decrtyp envelop key (envelop key is used to encrypt object data)
envelope_key_decrypt = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)
print(envelope_key_decrypt)

# download encrypted object directly to file (could also just write existing encrypted object to file)
s3.download_file(BUCKET, s3_key, filename)

# decrypt file
decrypt_file(envelope_key_decrypt['Plaintext'],filename,envelope_iv, int(original_size))

[tedder]

Brilliant. I just rewrote my code slightly and have it working. The main issue was the EncryptionContext- both what it needed to be, and that it needed to be json.parsed. It also means s3-encryption is unnecessary.

If you use download_file in your script, I'd suggest changing get_object to head_object since it isn't necessary.

[tedder]

Okay, I just reversed it- this code does a "put". It's my proof-of-concept code, so it needs some refactoring and such. I verified it works by pulling an object through the Java SDK, but please let me know if you test it independently.

https://github.com/tedder/s3-client-side-encryption/blob/master/put.py

[tedder]

Fixed an edge case with padding in my put.py, so make sure to update if you happen to have grabbed my older version.

I'd love to inject this into s3 similar to how s3transfer works but I need someone more fluent in boto3.

[tokenmathguy]

Checking in on this issue. I haven't seen traffic on it for a while. Does it make sense to revive it?

[Caligatio]

If I put together a PR that provides both AES-CBC and AES-GCM client side encryption, would there be appetite to adding it to the library?

[mariusgrigaitis]

I know that one of the reasons it might be hard to include it to boto3 is PyCrypto dependency. Maybe it could be made as separate package / optional dependency then?

[Caligatio]

@mariusgrigaitis One could use cryptography and avoid PyCrypto altogether. I have a proof-of-concept that uses it and it works just fine.

[mariusgrigaitis]

@Caligatio Could you maybe share it?

[rohitcelestial]

Isn't there a direct way to get decrypted from S3 by passing in KMS Key Id value?

[xor007]

No there is not.

[shaindyTIDE]

Hey Tedder,

Here is the code I got working to decrypt Java KMS client side uploads. Its hacky but was enough to prove out the solution. I did the encryption side for both python and dotnet as well if needed.

from __future__ import print_function
import boto
import boto3
import tempfile
import base64
import json
import Crypto
from Crypto.Cipher import AES
from Crypto import Random
import os, random, struct

# decrypt_file method from: http://eli.thegreenplace.net/2010/06/25/aes-encryption-of-files-in-python-with-pycrypto
def decrypt_file(key, in_filename, iv, original_size, out_filename=None, chunksize=16*1024):

    if not out_filename:
        out_filename = 'tempfile.png'

    with open(in_filename, 'rb') as infile:

        decryptor = AES.new(key, AES.MODE_CBC, iv)

        with open(out_filename, 'wb') as outfile:
            while True:
                chunk = infile.read(chunksize)
                if len(chunk) == 0:
                    break
                outfile.write(decryptor.decrypt(chunk))

            outfile.truncate(original_size)

REGION = '***'
BUCKET = '***'
s3_key = '***.png'
filename = '***.png'

s3 = boto3.client('s3')
kms = boto3.client('kms')

# download encrypted object from S3
encrypted = s3.get_object(Bucket=BUCKET,Key=s3_key)

# get object metadata from encrypted object and decode base64 strings
metadata = encrypted['Metadata']
envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encrypt_ctx = json.loads(metadata['x-amz-matdesc'])
original_size = metadata['x-amz-unencrypted-content-length']

# use AWS KMS to decrtyp envelop key (envelop key is used to encrypt object data)
envelope_key_decrypt = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)
print(envelope_key_decrypt)

# download encrypted object directly to file (could also just write existing encrypted object to file)
s3.download_file(BUCKET, s3_key, filename)

# decrypt file
decrypt_file(envelope_key_decrypt['Plaintext'],filename,envelope_iv, int(original_size))

Hi, you can send me the encryption side for python ?