aws_kms_encrypted_table.py giving "Failed to generate materials using AWS KMS"

[npatel1981]

Hi,

I created a sample encrypted table on AWS DynamoDB based on the example. In the table I created a partition_attribute and a sort_attribute. I also encrypted the table with KMS and then copied the KMS Master Key ARN to pass in to aws_cmk_id which is the argument in encrypt_item function in the python code. I am getting an error "Failed to generate materials using AWS KMS" am I doing something wrong? Do I need to do anything else to get this to work?

I have added the code I am trying to run below, which is basically the example and I just added a main function to pass in arguments.

# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
"""Example showing use of AWS KMS CMP with EncryptedTable."""
import os
import sys
import argparse
import getopt
import boto3
from boto3.dynamodb.types import Binary
import botocore.session

from dynamodb_encryption_sdk.encrypted.table import EncryptedTable
from dynamodb_encryption_sdk.identifiers import CryptoAction
from dynamodb_encryption_sdk.material_providers.aws_kms import AwsKmsCryptographicMaterialsProvider
from dynamodb_encryption_sdk.structures import AttributeActions

def encrypt_item(table_name, aws_cmk_id):
    """Demonstrate use of EncryptedTable to transparently encrypt an item."""
    index_key = {"partition_attribute": "is this", "sort_attribute": 55}
    plaintext_item = {
        "example": "data",
        "some numbers": 99,
        "and some binary": Binary(b"\x00\x01\x02"),
        "leave me": "alone",  # We want to ignore this attribute
    }
    # Collect all of the attributes that will be encrypted (used later).
    encrypted_attributes = set(plaintext_item.keys())
    encrypted_attributes.remove("leave me")
    # Collect all of the attributes that will not be encrypted (used later).
    unencrypted_attributes = set(index_key.keys())
    unencrypted_attributes.add("leave me")
    # Add the index pairs to the item.
    plaintext_item.update(index_key)

    session = boto3.Session(aws_access_key_id='xxxxxxxxxxxxxxxxxxxx',
                        aws_secret_access_key='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                        region_name='ca-central-1')

    dynamodb = session.resource('dynamodb')

    # Create a normal table resource.
    table = dynamodb.Table(table_name)  # generated code confuse pylint: disable=no-member
    # Create a crypto materials provider using the specified AWS KMS key.
    aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=aws_cmk_id)
    # Create attribute actions that tells the encrypted table to encrypt all attributes except one.
    actions = AttributeActions(
        default_action=CryptoAction.ENCRYPT_AND_SIGN, attribute_actions={"leave me": CryptoAction.DO_NOTHING}
    )
    # Use these objects to create an encrypted table resource.
    encrypted_table = EncryptedTable(table=table, materials_provider=aws_kms_cmp, attribute_actions=actions)

    # Put the item to the table, using the encrypted table resource to transparently encrypt it.
    encrypted_table.put_item(Item=plaintext_item)

    # Get the encrypted item using the standard table resource.
    encrypted_item = table.get_item(Key=index_key)["Item"]

    # Get the item using the encrypted table resource, transparently decyrpting it.
    decrypted_item = encrypted_table.get_item(Key=index_key)["Item"]

    # Verify that all of the attributes are different in the encrypted item
    for name in encrypted_attributes:
        assert encrypted_item[name] != plaintext_item[name]
        assert decrypted_item[name] == plaintext_item[name]

    # Verify that all of the attributes that should not be encrypted were not.
    for name in unencrypted_attributes:
        assert decrypted_item[name] == encrypted_item[name] == plaintext_item[name]

    # Clean up the item
    encrypted_table.delete_item(Key=index_key)

def main(argv):
    encrypt_item( 'TestTable', 'arn:aws:kms:ca-central-1:xxxxxxxxxxxx:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')

if __name__ == "__main__":
   main(sys.argv[1:])  

[robin-aws]

Hi there! Thanks for the reproduction code. While I see if I can reproduce the problem, I'd recommend double-checking that you're authorized to call the KMS GenerateDataKey and Decrypt operations: https://docs.aws.amazon.com/dynamodb-encryption-client/latest/devguide/direct-kms-provider.html

[tjmac21]

Having the same issue. Fixed this by accessing the protected botocore session within your session variable: session = boto3.Session(aws_access_key_id='xxxxxxxxxxxxxxxxxxxx', aws_secret_access_key='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', region_name='ca-central-1') aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=aws_cmk_id, botocore_session=session._session)

[mattsb42-aws]

@tjmac21 you hit the nail on the head.

I saw the same general shape as our example in your code, @npatel1981, and incorrectly assumed that the rest of it was our example. I missed where you created a custom boto3 session.

The issue here is that your DDB resource and the KMS client are using different credentials.

When you created a custom boto3 session with custom credentials:

session = boto3.Session(...)

Internally, that creates a custom botocore session, which is what contains the credentials.

When you then created the cryptographic materials provider without specifying a botocore session:

aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(...)

internally it created a default botocore session using whatever credentials are discovered in your environment[1].

The best solution to use the same credentials with both is to create a custom botocore session and use that for both the DDB resource and the AWS KMS crypto materials manager.

I do not recommend referencing the botocore session in the boto3 session as in @tjmac21's snippet because that relies on a _-prefixed variable, which is not part of the public API contract for boto3 sessions.

botocore_session = botocore.session.Session()
botocore_session.set_credentials(access_key="...", secret_key="...")

ddb_resource = boto3.session.Session(
    botocore_session=botocore_session,
    region_name="...",
)
aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(
    key_id="...",
    botocore_session=botocore_session,
)

[1] https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

[acioc]

Resolving per answer above. Please re-open this issue (or create a new one) if there are additional questions or you would like us to dive a bit deeper into anything.