search

aws / aws-cryptographic-material-providers-library-java

AWS Cryptographic Material Providers Library for Java
Apache License 2.0
4 stars 6 forks source link

"Invalid materials for decryption" error thrown on Encrypt operation. #144

Open mjsmcp opened 1 year ago

mjsmcp commented 1 year ago

There is an error message here indicates there is an error in decryption, but occurs during encryption. This led me to waste hours digging into the wrong part of my code.

Further, the error message itself contains no useful information. What makes the materials invalid? There are over a dozen different conditions in this method

WrapEdkMaterial:30, __default (EdkWrapping_Compile)
OnEncrypt_k:77, AwsKmsKeyring (AwsKmsKeyring_Compile)
OnEncrypt:20, _Companion_IKeyring (software.amazon.cryptography.materialproviders.internaldafny.types)
OnEncrypt:32, AwsKmsKeyring (AwsKmsKeyring_Compile)
OnEncrypt:53, Keyring (software.amazon.cryptography.materialproviders)
OnEncrypt:58, <redacted_class>$<redacted_class> (<redacted_package>)
OnEncrypt:91, Keyring$NativeWrapper (software.amazon.cryptography.materialproviders)
GetEncryptionMaterials_k:116, DefaultCMM (DefaultCMM_Compile)
GetEncryptionMaterials:20, _Companion_ICryptographicMaterialsManager (software.amazon.cryptography.materialproviders.internaldafny.types)
GetEncryptionMaterials:47, DefaultCMM (DefaultCMM_Compile)
GetEncryptionMaterials_k:106, RequiredEncryptionContextCMM (RequiredEncryptionContextCMM_Compile)
GetEncryptionMaterials:20, _Companion_ICryptographicMaterialsManager (software.amazon.cryptography.materialproviders.internaldafny.types)
GetEncryptionMaterials:49, RequiredEncryptionContextCMM (RequiredEncryptionContextCMM_Compile)
GetStructuredEncryptionMaterials:97, __default (AwsCryptographyDbEncryptionSdkStructuredEncryptionOperations_Compile)
EncryptStructure:375, __default (AwsCryptographyDbEncryptionSdkStructuredEncryptionOperations_Compile)
EncryptStructure:36, StructuredEncryptionClient (software.amazon.cryptography.dbencryptionsdk.structuredencryption.internaldafny)
EncryptItem:379, __default (AwsCryptographyDbEncryptionSdkDynamoDbItemEncryptorOperations_Compile)
EncryptItem:51, DynamoDbItemEncryptorClient (software.amazon.cryptography.dbencryptionsdk.dynamodb.itemencryptor.internaldafny)
Input:90, __default (PutItemTransform_Compile)
PutItemInputTransform:63, __default (AwsCryptographyDbEncryptionSdkDynamoDbTransformsOperations_Compile)
PutItemInputTransform:69, DynamoDbEncryptionTransformsClient (software.amazon.cryptography.dbencryptionsdk.dynamodb.transforms.internaldafny)
PutItemInputTransform:228, DynamoDbEncryptionTransforms (software.amazon.cryptography.dbencryptionsdk.dynamodb.transforms)
modifyRequest:121, DynamoDbEncryptionInterceptor (software.amazon.cryptography.dbencryptionsdk.dynamodb)
modifyRequest:68, ExecutionInterceptorChain (software.amazon.awssdk.core.interceptor)
runInitialInterceptors:181, AwsExecutionContextBuilder (software.amazon.awssdk.awscore.internal)
invokeInterceptorsAndCreateExecutionContext:117, AwsExecutionContextBuilder (software.amazon.awssdk.awscore.internal)
invokeInterceptorsAndCreateExecutionContext:69, AwsSyncClientHandler (software.amazon.awssdk.awscore.client.handler)
lambda$execute$1:78, BaseSyncClientHandler (software.amazon.awssdk.core.internal.handler)
get:-1, BaseSyncClientHandler$$Lambda$710/0x00007f73243d6e18 (software.amazon.awssdk.core.internal.handler)
measureApiCallSuccess:179, BaseSyncClientHandler (software.amazon.awssdk.core.internal.handler)
execute:76, BaseSyncClientHandler (software.amazon.awssdk.core.internal.handler)
execute:45, SdkSyncClientHandler (software.amazon.awssdk.core.client.handler)
execute:56, AwsSyncClientHandler (software.amazon.awssdk.awscore.client.handler)
putItem:3926, DefaultDynamoDbClient (software.amazon.awssdk.services.dynamodb)
apply:-1, PutItemOperation$$Lambda$1471/0x00007f73244ffb78 (software.amazon.awssdk.enhanced.dynamodb.internal.operations)
execute:114, CommonOperation (software.amazon.awssdk.enhanced.dynamodb.internal.operations)
executeOnPrimaryIndex:59, TableOperation (software.amazon.awssdk.enhanced.dynamodb.internal.operations)
putItem:255, DefaultDynamoDbTable (software.amazon.awssdk.enhanced.dynamodb.internal.client)
<redacted_method>:64, <redacted_class> (<mypackage>)
<redacted_method>:52, <redacted_class> (<mypackage>)
<redacted_method>:56, <redacted_class>$<redacted_class> (<mypackage>)
invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:77, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
invoke:568, Method (java.lang.reflect)
invokeMethod:728, ReflectionUtils (org.junit.platform.commons.util)
proceed:60, MethodInvocation (org.junit.jupiter.engine.execution)
proceed:131, InvocationInterceptorChain$ValidatingInvocation (org.junit.jupiter.engine.execution)
intercept:156, TimeoutExtension (org.junit.jupiter.engine.extension)
interceptTestableMethod:147, TimeoutExtension (org.junit.jupiter.engine.extension)
interceptTestMethod:86, TimeoutExtension (org.junit.jupiter.engine.extension)
apply:-1, TestMethodTestDescriptor$$Lambda$221/0x00007f7324134400 (org.junit.jupiter.engine.descriptor)
lambda$ofVoidMethod$0:103, InterceptingExecutableInvoker$ReflectiveInterceptorCall (org.junit.jupiter.engine.execution)
apply:-1, InterceptingExecutableInvoker$ReflectiveInterceptorCall$$Lambda$222/0x00007f7324134820 (org.junit.jupiter.engine.execution)
lambda$invoke$0:93, InterceptingExecutableInvoker (org.junit.jupiter.engine.execution)
apply:-1, InterceptingExecutableInvoker$$Lambda$405/0x00007f7324161c08 (org.junit.jupiter.engine.execution)
proceed:106, InvocationInterceptorChain$InterceptedInvocation (org.junit.jupiter.engine.execution)
proceed:64, InvocationInterceptorChain (org.junit.jupiter.engine.execution)
chainAndInvoke:45, InvocationInterceptorChain (org.junit.jupiter.engine.execution)
invoke:37, InvocationInterceptorChain (org.junit.jupiter.engine.execution)
invoke:92, InterceptingExecutableInvoker (org.junit.jupiter.engine.execution)
invoke:86, InterceptingExecutableInvoker (org.junit.jupiter.engine.execution)
lambda$invokeTestMethod$7:218, TestMethodTestDescriptor (org.junit.jupiter.engine.descriptor)
execute:-1, TestMethodTestDescriptor$$Lambda$1403/0x00007f73244ed028 (org.junit.jupiter.engine.descriptor)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
invokeTestMethod:214, TestMethodTestDescriptor (org.junit.jupiter.engine.descriptor)
execute:139, TestMethodTestDescriptor (org.junit.jupiter.engine.descriptor)
execute:69, TestMethodTestDescriptor (org.junit.jupiter.engine.descriptor)
lambda$executeRecursively$6:151, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$313/0x00007f7324147c50 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$8:141, NodeTestTask (org.junit.platform.engine.support.hierarchical)
invoke:-1, NodeTestTask$$Lambda$312/0x00007f7324147a28 (org.junit.platform.engine.support.hierarchical)
around:137, Node (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$9:139, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$311/0x00007f7324147600 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
executeRecursively:138, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:95, NodeTestTask (org.junit.platform.engine.support.hierarchical)
accept:-1, SameThreadHierarchicalTestExecutorService$$Lambda$339/0x00007f7324156b10 (org.junit.platform.engine.support.hierarchical)
forEach:1511, ArrayList (java.util)
invokeAll:41, SameThreadHierarchicalTestExecutorService (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$6:155, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$313/0x00007f7324147c50 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$8:141, NodeTestTask (org.junit.platform.engine.support.hierarchical)
invoke:-1, NodeTestTask$$Lambda$312/0x00007f7324147a28 (org.junit.platform.engine.support.hierarchical)
around:137, Node (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$9:139, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$311/0x00007f7324147600 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
executeRecursively:138, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:95, NodeTestTask (org.junit.platform.engine.support.hierarchical)
accept:-1, SameThreadHierarchicalTestExecutorService$$Lambda$339/0x00007f7324156b10 (org.junit.platform.engine.support.hierarchical)
forEach:1511, ArrayList (java.util)
invokeAll:41, SameThreadHierarchicalTestExecutorService (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$6:155, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$313/0x00007f7324147c50 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$8:141, NodeTestTask (org.junit.platform.engine.support.hierarchical)
invoke:-1, NodeTestTask$$Lambda$312/0x00007f7324147a28 (org.junit.platform.engine.support.hierarchical)
around:137, Node (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$9:139, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$311/0x00007f7324147600 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
executeRecursively:138, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:95, NodeTestTask (org.junit.platform.engine.support.hierarchical)
accept:-1, SameThreadHierarchicalTestExecutorService$$Lambda$339/0x00007f7324156b10 (org.junit.platform.engine.support.hierarchical)
forEach:1511, ArrayList (java.util)
invokeAll:41, SameThreadHierarchicalTestExecutorService (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$6:155, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$313/0x00007f7324147c50 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$8:141, NodeTestTask (org.junit.platform.engine.support.hierarchical)
invoke:-1, NodeTestTask$$Lambda$312/0x00007f7324147a28 (org.junit.platform.engine.support.hierarchical)
around:137, Node (org.junit.platform.engine.support.hierarchical)
lambda$executeRecursively$9:139, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:-1, NodeTestTask$$Lambda$311/0x00007f7324147600 (org.junit.platform.engine.support.hierarchical)
execute:73, ThrowableCollector (org.junit.platform.engine.support.hierarchical)
executeRecursively:138, NodeTestTask (org.junit.platform.engine.support.hierarchical)
execute:95, NodeTestTask (org.junit.platform.engine.support.hierarchical)
submit:35, SameThreadHierarchicalTestExecutorService (org.junit.platform.engine.support.hierarchical)
execute:57, HierarchicalTestExecutor (org.junit.platform.engine.support.hierarchical)
execute:54, HierarchicalTestEngine (org.junit.platform.engine.support.hierarchical)
execute:108, EngineExecutionOrchestrator (org.junit.platform.launcher.core)
execute:88, EngineExecutionOrchestrator (org.junit.platform.launcher.core)
lambda$execute$0:54, EngineExecutionOrchestrator (org.junit.platform.launcher.core)
accept:-1, EngineExecutionOrchestrator$$Lambda$286/0x00007f732413cd40 (org.junit.platform.launcher.core)
withInterceptedStreams:67, EngineExecutionOrchestrator (org.junit.platform.launcher.core)
execute:52, EngineExecutionOrchestrator (org.junit.platform.launcher.core)
execute:96, DefaultLauncher (org.junit.platform.launcher.core)
execute:75, DefaultLauncher (org.junit.platform.launcher.core)
processAllTestClasses:99, JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor (org.gradle.api.internal.tasks.testing.junitplatform)
access$000:79, JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor (org.gradle.api.internal.tasks.testing.junitplatform)
stop:75, JUnitPlatformTestClassProcessor (org.gradle.api.internal.tasks.testing.junitplatform)
stop:61, SuiteTestClassProcessor (org.gradle.api.internal.tasks.testing)
invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:77, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
invoke:568, Method (java.lang.reflect)
dispatch:36, ReflectionDispatch (org.gradle.internal.dispatch)
dispatch:24, ReflectionDispatch (org.gradle.internal.dispatch)
dispatch:33, ContextClassLoaderDispatch (org.gradle.internal.dispatch)
invoke:94, ProxyDispatchAdapter$DispatchingInvocationHandler (org.gradle.internal.dispatch)
stop:-1, $Proxy5 (jdk.proxy2)
run:193, TestWorker$3 (org.gradle.api.internal.tasks.testing.worker)
executeAndMaintainThreadName:129, TestWorker (org.gradle.api.internal.tasks.testing.worker)
execute:100, TestWorker (org.gradle.api.internal.tasks.testing.worker)
execute:60, TestWorker (org.gradle.api.internal.tasks.testing.worker)
execute:56, ActionExecutionWorker (org.gradle.process.internal.worker.child)
call:133, SystemApplicationClassLoaderWorker (org.gradle.process.internal.worker.child)
call:71, SystemApplicationClassLoaderWorker (org.gradle.process.internal.worker.child)
run:69, GradleWorkerMain (worker.org.gradle.process.internal.worker)
main:74, GradleWorkerMain (worker.org.gradle.process.internal.worker)
justplaz commented 1 year ago

Hi there,

Thanks for reaching out!

First, I'd like to apologize for the misleading error message. That is a typo, and I have a PR out to fix the error text: https://github.com/aws/aws-cryptographic-material-providers-library-dafny/pull/84.

Next, to help with debugging, could you please include a sample of the code which generated this error? Note that if you are using the RequiredEncryptionContextCMM, you MUST provide the required encryption context keys for each request.

mjsmcp commented 1 year ago

Thank you for fixing that one error message, though. Hopefully it'll save someone else a few hours.

I'm not entirely sure the issue is with the SDK. The code works as expected when using KMS proper, but I get these errors when I use it with this kms local library.

Without more useful error messages or logging, though, I've run into a wall trying to figure out what's happening. Dafny doesn't seem well-supported by the IntelliJ debugger (all I found was this 4 year old dodgy-looking plugin).

This is more of a developer experience request, as logging or some more specific errors from the dafny module would help a lot in trying to figure this out. (As would an official KMS local, just saying 😉) A giant wall of boolean checks with no detail is very difficult to work with when things like this go wrong.

As for the code, here are some snippets.

KmsModule (Guice Injection)

    @Provides
    @Singleton
    @Named("DatabaseEncryptionKeyId")
    public String provideDatabaseEncryptionKeyId(KmsClient kmsClient) {
        return System.getenv().get("DB_KEY_ID");
    }

    @Provides
    @Singleton
    public KmsClient provideKmsClient() {
        return KmsClient.create();
    }

    @Provides
    @Singleton
    public IKeyring provideKeyring(KmsClient kmsClient, @Named("DatabaseEncryptionKeyId") String keyId) {
        final MaterialProviders matProv = MaterialProviders.builder()
                .MaterialProvidersConfig(MaterialProvidersConfig.builder().build())
                .build();
        CreateAwsKmsKeyringInput ki = CreateAwsKmsKeyringInput.builder()
                .kmsClient(kmsClient)
                .kmsKeyId(keyId)
                .build();

        return matProv.CreateAwsKmsKeyring(ki);
    }

DynamoModule (Guice Injection)

    @Provides
    @Singleton
    public Map<String, DynamoDbEnhancedTableEncryptionConfig> provideTableConfigs(IKeyring keyring) {
        return Map.of(
                TableSchemas.REDACTED1_TABLE_NAME, DynamoDbEnhancedTableEncryptionConfig.builder()
                        .logicalTableName(TableSchemas.REDACTED1_TABLE_NAME)
                        .algorithmSuiteId(DBEAlgorithmSuiteId.ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY_ECDSA_P384_SYMSIG_HMAC_SHA384)
                        .schemaOnEncrypt(TableSchemas.REDACTED1_TABLE_SCHEMA)
                        .keyring(keyring)
                        .build(),
                TableSchemas.REDACTED2_TABLE_NAME, DynamoDbEnhancedTableEncryptionConfig.builder()
                        .logicalTableName(TableSchemas.REDACTED2_TABLE_NAME)
                        .algorithmSuiteId(DBEAlgorithmSuiteId.ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY_ECDSA_P384_SYMSIG_HMAC_SHA384)
                        .schemaOnEncrypt(TableSchemas.REDACTED2_TABLE_SCHEMA)
                        .keyring(keyring)
                        .build()
        );
    }

    @Provides
    @Singleton
    @Named("DynamoDbEncryptionInterceptor")
    public ExecutionInterceptor provideDynamoDbEncryptionInterceptor(Map<String, DynamoDbEnhancedTableEncryptionConfig> tableConfigs) {
        return DynamoDbEnhancedClientEncryption.CreateDynamoDbEncryptionInterceptor(
                CreateDynamoDbEncryptionInterceptorInput.builder()
                        .tableEncryptionConfigs(tableConfigs)
                        .build()
        );
    }

    @Provides
    @Singleton
    public DynamoDbClient provideDynamoClient(@Named("DynamoDbEncryptionInterceptor") ExecutionInterceptor encryptionInterceptor, DynamoDbEndpointProvider endpointProvider,
                                              Region region, AwsCredentialsProvider credentialsProvider) {
        return DynamoDbClient.builder()
                .endpointProvider(endpointProvider)
                .credentialsProvider(credentialsProvider)
                .region(region)
                .overrideConfiguration(
                        ClientOverrideConfiguration.builder()
                                .addExecutionInterceptor(encryptionInterceptor)
                                .build()
                ).build();
    }

    @Provides
    @Singleton
    public DynamoDbEnhancedClient provideDynamoEnhancedClient(DynamoDbClient dynamoDbClient) {
        return DynamoDbEnhancedClient.builder()
                .dynamoDbClient(dynamoDbClient)
                .build();
    }

And then my TableSchemas have .addTag(new SignOnlyTag()) as needed on the model.

seebees commented 1 year ago

Hi,

Better stack traces is something we are working on.

We have not had any requests for mocks of this kind before. Why do you want to run this kind of testing?

mjsmcp commented 1 year ago

I prefer testing as much of my software before commit as possible, and using mock services enables that.

Encrypting customer data is an important element of what I'm building, so I was hoping to be able to verify that with a mock KMS locally, hence trying the kms-local library. I can always hack something together with Mockito, but then I'm relying on my interpretation of behavior to be correct instead of something vended that is vetted by other users.

As for the stack traces, is there a ticket I can follow for that besides this one? I'm at a dead end until I can get better error messaging or better debugger support for dafny.

lucasmcdonald3 commented 11 months ago

Hi @mjsmcp,

At this time, we do not support any libraries that mock AWS services.

To solve your immediate requirement, I would suggest creating a new KMS key strictly for testing use. This would let you validate the true behavior of your application, as you are interacting with real KMS interfaces.

With respect to better stack traces: We do not have an external-facing ticket or date at this time, but it is on our roadmap.

Best, Lucas