Problem with CMK in AWS when DynamoDB is replicating to new region

[nan008]

This is tying ti issue #221

We are creating DR and replicating to another region within DynamoDB feature for global tables. The problem is with the key that you are asking to create as part of the setup.

When you cross to another region the tables are there but the key cannot be replicated. So the global table are useless as you need the key from previous region to decrypt

Any solution to that? the issue 221 was sent in 2018, so hopefully we will have more info about this problem

[nan008]

I do not know if this project is still managed but I just commented on another issue that using exported keys in KMS doesn't work at least with different key then Credstash (did not tried with credstash as it takes 7 days to delete the key from KMS)

I have exported the same key into two KMS regions - N. Virginia and Oregon called cred_test Created global table for testing https://github.com/fugue/credstash/issues/188 when OP is using alias of the key

credstash -p dpe_non_prod put --key cloudwatch_feeder testme testvalue

I cannot do the same as my

➜ venv AWS_DEFAULT_REGION=us-east-1 credstash -t stage putall @stage2.json -v 0000000000000000002 --key cred_test

results in

Error on credstash 1.16.1

KMS ERROR: Could not generate key using KMS key cred_test (Details: An error occurred (NotFoundException) when calling the GenerateDataKey operation: Invalid keyId cred_test) KMS ERROR: Could not generate key using KMS key cred_test (Details: An error occurred (NotFoundException) when calling the GenerateDataKey operation: Invalid keyId cred_test)

Error on credstash 1.14.0

Traceback (most recent call last): File "/Users/nan/myprojects/venv/bin/credstash.py", line 78, in generate_key_data KeyId=self.key_id, EncryptionContext=self.encryption_context, NumberOfBytes=number_of_bytes File "/Users/nan/myprojects/venv/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/nan/myprojects/venv/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.NotFoundException: An error occurred (NotFoundException) when calling the GenerateDataKey operation: Invalid keyId cred_test

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/nan/myprojects/venv/bin/credstash", line 8, in <module> sys.exit(main()) File "/Users/nan/myprojects/venv/bin/credstash.py", line 919, in main putAllSecretsAction(args, region, **session_params) File "/Users/nan/myprojects/venv/bin/credstash.py", line 245, in func_wrapper return func(*args, **kwargs) File "/Users/nan/myprojects/venv/bin/credstash.py", line 414, in putAllSecretsAction putSecretAction(args, region, **session_params) File "/Users/nan/myprojects/venv/bin/credstash.py", line 245, in func_wrapper return func(*args, **kwargs) File "/Users/nan/myprojects/venv/bin/credstash.py", line 390, in putSecretAction **session_params): File "/Users/nan/myprojects/venv/bin/credstash.py", line 296, in putSecret digest_method=digest, File "/Users/nan/myprojects/venv/bin/credstash.py", line 616, in seal_aes_ctr_legacy key, encoded_key = key_service.generate_key_data(64) File "/Users/nan/myprojects/venv/bin/credstash.py", line 81, in generate_key_data raise KmsError("Could not generate key using KMS key %s (Detail: %s)" % (self.key_id, e.message)) AttributeError: 'NotFoundException' object has no attribute 'message'

When I use ARN of the key it is working and putting the secrets into the table

➜ venv AWS_DEFAULT_REGION=us-east-1 credstash -t stage putall @stage2.json -v 0000000000000000002 --key arn:aws:kms:us-east-1:123456789:key/1234-5678-9101-t1121-13141516 TEST_KEY has been stored SOMETHING_KEY has been stored ELSE_KEY has been stored

PLEASE NOTE: the same external key is imported to the KMS but it has different wrapper and import token (will this work with name credstash?) and getting the secret from the global table but on the Oregon side

➜ venv AWS_DEFAULT_REGION=us-west-2 credstash -t stage get TEST_KEY KMS ERROR: Decryption error An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

ON A DIFFERENT NOTE: Issues with 1.15 and 1.16.1 regarding putall/get all from json as well like in https://github.com/fugue/credstash/issues/211 (1.16.1 is working on venv on Mac but not on linux)

1.14.0 is working without issues so I have to downgrade, upgrade to test with credstash controller.

[nan008]

I can now confirm that having the same external key imported to KMS in two different regions with the name credstash is NOT WORKING. Tried versions of credstash 1.14.0 and 1.16.1

My two test regions Ohio us-east-2 Oregon us-west-2

Created Global table in Oregon with added region in Ohio

I run in Oregon with KMS holding the external key named credstash

(venv)➜ AWS_DEFAULT_REGION=us-west-2 credstash -t test8 putall @test2.json -v 0000000000000000002 KEY_KEY has been stored TEST_KEY has been stored SOME_KEY has been stored

AWS_DEFAULT_REGION=us-west-2 credstash -t table8 get KEY_KEY testingtestingtesting

So no problem there

I mirror the steps of importing the key to KMS in Ohio the SAME key but different wrapper and import token as those are created by AWS by region

the table8 replicated in DynamoDB which is confirmed by running list command

AWS_DEFAULT_REGION=us-east-2 credstash -t table8 list

KEY_KEY -- version 0000000000000000002 TEST_KEY -- version 0000000000000000002 SOME_KEY -- version 0000000000000000002

Yet running this command in Ohio (REMINDER: I have imported the SAME key with the name credstash just different wrapper and import token)

➜ venv AWS_DEFAULT_REGION=us-east-2 credstash -t table8 get KEY_KEY

gives me

KMS ERROR: Decryption error An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

[mike-luminal]

Thanks for the input, I will start looking at this for the next release.

[mike-luminal]

One workaround is to set a master KMS region that remains the same even when the DynamoDB table is in different regions.

[mike-luminal]

@nan008 I've created a PR with a potential fix for your use-case, can you take a look and let me know if that will suffice?