Customer provided (and managed) key?

[lsong0905]

Hello,

As described in the following links

1. https://docs.microsoft.com/en-us/azure/storage/common/storage-service-encryption?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
2. https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-customer-provided-key

I want to apply and test operations such as upload, download, etc with customer provided key (and customer managed key as well). However, I am having trouble finding a way to provide the key to the sdk unless I am not missing something. I see blob.h has some functions and flags that seem to enable server side encryption such as set_server_encrypted() and server_encrypted(), but no functions to receive the key, specifying the algorithm, its SHA256 value, etc.

Would you please guide me how to apply SSE with customer provided (and managed) key for upload operation?

The sdk version is azure-storage-cpp-7.0.0.

[Jinming-Hu]

Hi @lsong0905 , the customer provided key feature is not implemented in this cpp sdk yet. We can provide in future release if you need.

To implement the customer provided key feature, we need to add a few APIs, so that you can provide the key to server side for encryption when uploading and for decryption when downloading.

And about the customer managed key, our cpp sdk doesn't need to do anything, you can just configure it on Azure Portal.

There's another feature with which you can store the key somewhere and reference it with a name rather than provide it in plaintext everytime you operate on a blob. But I think this feature is not completely ready on server side.

I want to know which is the feature you want, so that we can help accordingly.

[Jinming-Hu]

@lsong0905 In addition, about the server_encrypted property of a blob, now newly uploaded blobs are encrypted by default, by Microsoft managed key or customer managed key or other customer provided keys. So for new blobs, this property is always true. Some time ago when server-side encryption is not turned on by default, this property for some blobs might be false. You cannot edit this property anyway.

[lsong0905]

@JinmingHu-MSFT Thank you for the reply! Regarding the customer-provided key, can we still specify this in the request header as mentioned here (https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob)? I do not see the request headers for key, algorithm and sha256 defined in constants.dat, but still wonder if it is possible to add them to the header.

Also, I have a few question regarding the customer-managed key though. As you mentioned, I believe Azure storage uses Microsoft-managed keys to encrypt by default, and this is turned on by default as well. Let's say I want to configure this encryption with customer-managed keys via SDK (application level), and I have a new key in the key vault that I want to use to encrypt. How can I upload blobs with this key via SDK, for example, so that SDK receives the key as customer-managed key, and encrypt blobs with it.

[Jinming-Hu]

@lsong0905

I do not see the request headers for key, algorithm and sha256 defined in constants.dat, but still wonder if it is possible to add them to the header.

As I said in the previous reply, we haven't implemented this feature in current release. If you request it, we'll add it in future release, maybe 7.3.0 I think.

As you can see in the link you mentioned, the encryption key is provided in plaintext in header x-ms-encryption-key. Let's call the feature above CPK.

Now there is another new feature, which I think is not completely ready at server side, called CPK-N. In this feature, you store a key in Microsoft Key Vault or somewhere else, and associate it with a name, like key1, key2, key3, etc. Instead of providing the key in plaintext everytime you call an API, you just reference it by its name, in header x-ms-encryption-key-name. Also in this feature, there is going to be a new function, which is that you can set a container-scoped default key. This container-scoped key overrides storage-account-scoped key and can be overridden by per-blob key provided in header x-ms-encryption-key or x-ms-encryption-key-name. So basically, you get more granular control with this CPK-N feature.

So which is the feature you really want, CPK or CPK-N, or both?

[Jinming-Hu]

@lsong0905 Customer-managed key is a storage-account-scoped setting, you can easily configure it on Azure Portal. Once configured, all new blobs in this storage account are encrypted with the new key instead of the default Microsoft-managed key. Neither our cpp sdk nor your code calling cpp sdk needs to do anything.

If you want to configure this via SDK rather than Azure Portal, have a look at SRP SDK here in the encryption seciton.

[lsong0905]

@JinmingHu-MSFT Thank you for the reply. To answer your question regarding customer provided key, I would say both CPK and CPK-N would be okay with me. My initial plan was CPK. And I know the feature has not been implemented yet as you mentioned. However, if I add optional headers (x-ms-encryption-key, x-ms-encryption-key-sha256 and x-ms-encryption-algorithm) and provide them in plaintext for the REST requests, is it going to apply encryption with the provided headers?

And yes. I will reference the link to configure the setting. Thank you.

[Jinming-Hu]

Hi @lsong0905 ,

if I add optional headers (x-ms-encryption-key, x-ms-encryption-key-sha256 and x-ms-encryption-algorithm) and provide them in plaintext for the REST requests, is it going to apply encryption with the provided headers?

Yes, it's going to work, since the CPK feature is ready at server side long time ago, and it's not complicated, just three new headers. Maybe you need to edit the source code and build the library by yourself.

[lsong0905]

@JinmingHu-MSFT Thank you for the help. I will try with CPK and see if I get expected results. It would be nice to have that implemented in future release as well. Plus, is CPK-N will be fully available anytime near future?

[Jinming-Hu]

@lsong0905 Yes, CPK-N is around the corner, do you want it in future release as well?

[lsong0905]

@JinmingHu-MSFT Yes. That would give us more options for server side encryption.

Regarding CPK, I wrote a simple test program and tried uploading blobs with CPK. I am sure I am missing something or probably adding the headers in wrong way, but when I check the uploaded blob, the blob has been uploaded but was NOT encrypted.

Here is a code snippet from my test program.

...
operation_context m_context;
web::http::http_headers& myHeader = m_context.user_headers();
myHeader.add(_XPLATSTR("x-ms-encryption-key"), _XPLATSTR("at1TMx82nEy7SoAK8jHYanMQDVZMSLayXaaUvTc6CP1="));
myHeader.add(_XPLATSTR("x-ms-encryption-key-sha256"), _XPLATSTR("ICbcyjSQU33x7z+RXeHISnc8GqKpVEZyR9BUJNkpcg4"));
myHeader.add(_XPLATSTR("x-ms-encryption-algorithm"), _XPLATSTR("AES256"));
...

block_blob.upload_from_stream((*iface->m_istream), olen, azure::storage::access_condition(), reqOptions, m_context);`

From the azure logging file, I see the request header contains the three headers I added for encryption.

18 2020-02-12 13:59:31 FF34D565-6ACA-49C8-962A-832699714AF2 : StringToSign: PUT...65536.........x-ms-client-request-id:FF34D565-6ACA-49C8-962A-832699714AF2.x-ms-date:Wed, 12 Feb 2020 21:59:31 GMT.x-ms-encryption-algorithm:AES256.x-ms-encryption-key:at1TMx82nEy7SoAK8jHYanMQDVZMSLayXaaUvTc6CP1=.x-ms-encryption-key-sha256:ICbcyjSQU33x7z+RXeHISnc8GqKpVE ZyR9BUJNkpcg4.x-ms-version:2019-02-02./devstoreaccount1/devstoreaccount1/leoTest/Encrypted_TestObject.0.blockid:MUZCNDdFNTctMTk5RS00MTkxLTkwMzEtNEQwRDczQUUzNzkyLTAwMDAwMA==.c omp:block.timeout:300

When I ran list command via azure cli, I see the object I uploaded, but its "serverEncrypted" field is set to false, and I was able to download without providing the key. So I assume the blob was not encrypted.

"content": null, "deleted": false, "metadata": null, "name": "Encrypted_TestObject.0", "properties": { "appendBlobCommittedBlockCount": null, "blobTier": null, "blobTierChangeTime": null, "blobTierInferred": false, "blobType": "BlockBlob", "contentLength": 65536, "contentRange": null, "contentSettings": { "cacheControl": null, "contentDisposition": null, "contentEncoding": null, "contentLanguage": null, "contentMd5": "RtSczKvyuiqUQe255D5gRw==", "contentType": "application/octet-stream" }, "copy": { "completionTime": null, "id": null, "progress": null, "source": null, "status": null, "statusDescription": null }, "deletedTime": null, "etag": "oGyKRjAWIgW1QSBK3zEXn5cgV5U", "lastModified": "2020-02-12T21:58:34+00:00", "lease": { "duration": null, "state": "available", "status": "unlocked" }, "pageBlobSequenceNumber": null, "remainingRetentionDays": null, "sequenceNumber": null, "serverEncrypted": false }, "snapshot": null

Would you please guide me how to add the additional headers into the request?

[Jinming-Hu]

@lsong0905 Your value of header x-ms-encryption-key-sha256 is wrong. Result of SHA-256 should be 32 bytes (256bits), after base64 encoding, it should be ceil(32/3)*4=44bytes.

Try kyj0Rh7LdcvES0oFmh6ug/u+vuY7vKv0MHdU705QMOA=

[lsong0905]

@JinmingHu-MSFT I still see the same symptom with the value of sha256. The blob has been uploaded without encryption, and I am able to download it without the key.

18 2020-02-12 20:43:13 72603ABD-282C-4E1F-A772-4779C2DF9B54 : StringToSign: PUT...65536.........x-ms-client-request-id:72603ABD-282C-4E1F-A772-4779C2DF9B54.x-ms-date:Thu, 13 Feb 2020 04:43:13 GMT .x-ms-encryption-algorithm:AES256.x-ms-encryption-key:at1TMx82nEy7SoAK8jHYanMQDVZMSLayXaaUvTc6CP1=.x-ms-encryption-key-sha256:kyj0Rh7LdcvES0oFmh6ug/u+vuY7vKv0MHdU705QMOA=.x-ms-version:2019-02- 02./devstoreaccount1/devstoreaccount1/leoTest/Encrypted_TestObject.0.blockid:MTU2NEE1MTMtODI1Ni00NTYxLTlEQ0YtMEVBMUZFNEIyQ0UxLTAwMDAwMA==.comp:block.timeout:300 19 2020-02-12 20:43:13 72603ABD-282C-4E1F-A772-4779C2DF9B54 : Response received. Status code = 201. Reason = Created 20 2020-02-12 20:43:13 72603ABD-282C-4E1F-A772-4779C2DF9B54 : Successful request ID = 350d2980-4e1b-11ea-8d1b-bfbcf570bd45 21 2020-02-12 20:43:13 72603ABD-282C-4E1F-A772-4779C2DF9B54 : Operation completed successfully 22 2020-02-12 20:43:13 72603ABD-282C-4E1F-A772-4779C2DF9B54 : This operation can only be executed against the primary storage location.

[Jinming-Hu]

@lsong0905 It works on my side.

Here's my test code

        azure::storage::cloud_storage_account storage_account = azure::storage::cloud_storage_account::parse(_XPLATSTR("CONNECTION_STRING"));
        azure::storage::cloud_blob_client blob_client = storage_account.create_cloud_blob_client();
        azure::storage::cloud_blob_container container = blob_client.get_container_reference(_XPLATSTR("container1"));

        azure::storage::cloud_block_blob blob1 = container.get_block_blob_reference(_XPLATSTR("eblob"));
        std::vector<uint8_t> buffer;
        size_t buffer_size = 128 * 1024;
        buffer.resize(buffer_size);
        concurrency::streams::container_buffer<std::vector<uint8_t>> input_buffer(buffer);
        azure::storage::operation_context context;
        context.user_headers().add(_XPLATSTR("x-ms-encryption-key"), _XPLATSTR("at1TMx82nEy7SoAK8jHYanMQDVZMSLayXaaUvTc6CP1="));
        context.user_headers().add(_XPLATSTR("x-ms-encryption-key-sha256"), _XPLATSTR("kyj0Rh7LdcvES0oFmh6ug/u+vuY7vKv0MHdU705QMOA="));
        context.user_headers().add(_XPLATSTR("x-ms-encryption-algorithm"), _XPLATSTR("AES256"));

        blob1.upload_from_stream(input_buffer, buffer_size, azure::storage::access_condition(), azure::storage::blob_request_options(), context);

        concurrency::streams::container_buffer<std::vector<uint8_t>> output_buffer;
        concurrency::streams::ostream output_stream(output_buffer);
        try {
            blob1.download_to_stream(output_stream);
        }
        catch (azure::storage::storage_exception& e) {
            std::cout << e.what() << std::endl;
        }

        blob1.download_to_stream(output_stream, azure::storage::access_condition(), azure::storage::blob_request_options(), context);
        std::cout << "server encrypted: " << blob1.properties().server_encrypted() << std::endl;

Can you check your code against my code and see if there's anything wrong?

[lsong0905]

@JinmingHu-MSFT I do not think my code differs much from what you have provided. Is there some change needed to be made in the sdk (7.0.0) before running the code? One thing I see different is that I get null (or false) from server_encrypted().

Server encrypted: (null)

[lsong0905]

Update: I was able to apply SSE with CPK without the issue if connected to real backend. The problem is only happening if I connect and run operations via emulator.

[Jinming-Hu]

@lsong0905 Seems the emulator doesn't function well

[lsong0905]

@JinmingHu-MSFT Thank you for the help! How can I track of the new features (CPK and CPK-N)?

[Jinming-Hu]

@lsong0905 The CPK is done, see it here. It will be merged into master branch soon after code review. The CPK-N might still take some time, I'll update this issue if there are any updates.

[Jinming-Hu]

Update: I was able to apply SSE with CPK without the issue if connected to real backend. The problem is only happening if I connect and run operations via emulator.

BTW, which emulator are you using, Azurite?

[lsong0905]

@JinmingHu-MSFT Thank you for the link! And yes. It is Azurite.

[Jinming-Hu]

@lsong0905 I just confirmed with my colleague, Azurite doesn't support CPK-V yet. If you need that, you may want to create a feature request on their GitHub repo.

[lsong0905]

@JinmingHu-MSFT Okay! I am now testing against the real backend, and if our organization needs to test with Azurite, I will create a feature request on the repo.