Presence of external DynamoDbAsyncClient leads to unexpected DynamoDbLockRepository and ConcurrentMetadataStore creation

[alexkitts]

In what version(s) of Spring Cloud Stream Binder for AWS Kinesis are you seeing this issue?

For example: 4.0.0

Describe the bug

Our project makes use of a DynamoDbAsyncClient in an environment where tables and AWS roles are tightly controlled. The present of the DynamoDbAsyncClient seems to trigger the @CondtionalOnBean annotations here:

https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/blob/4098381e1b0cdfc7964f695c6682f9bc6532ebbc/spring-cloud-stream-binder-kinesis/src/main/java/org/springframework/cloud/stream/binder/kinesis/config/KinesisBinderConfiguration.java#L154C21-L154C40

This means that where we would expect the DynamoDbLockRepository and ConcurrentMetadataStore to be NullBeans, we get genuine articles which subsequently try and create the corresponding dynamo tables. This leads to this error on startup:

{"timestamp":"2023-09-29T07:05:42.036Z","level":"ERROR","thread":"sdk-async-response-3-0","logger":"org.springframework.integration.aws.metadata.DynamoDbMetadataStore","message":"Cannot create DynamoDb table: SpringIntegrationMetadataStore","context":"default","exception":"software.amazon.awssdk.services.dynamodb.model.DynamoDbException: User: arn:aws:sts<REDACTED>:assumed-role/<REDACTED>/aws-sdk-java-1695971140247 is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:eu-west-2:<REDACTED>:table/SpringIntegrationMetadataStore because no identity-based policy allows the dynamodb:DescribeTable action...

This can be overcome using mocked versions of the ConcurrentMetadataStore and DynamoDbLockRepository as described here: https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/93 but that is hardly ideal.

To Reproduce

Create new project, setup binder for production of events, provide external @Bean DynamoDbAsyncClient and then see how the other discussed beans try to (or maybe succeed in) creating unexpected tables at startup.

Expected behavior

The existence of an external DynamoDbAsyncClient should not trigger the creation of unexpected dynamo tables.

Thanks!

[artembilan]

If you don't need a DynamoDbMetadataStore, then you really have to use a mocked variant for the ConcurrentMetadataStore., for example an in-memory SimpleMetadataStore. If you still need a DynamoDbMetadataStore, but you don't want it to create table, then have that table already created upfront. The logic there is like this:

this.dynamoDB.describeTable(request -> request.tableName(this.tableName))
                .thenRun(() -> { })
                .exceptionallyCompose((ex) -> {
                    Throwable cause = ex.getCause();
                    if (cause instanceof ResourceNotFoundException) {
                        if (logger.isInfoEnabled()) {
                            logger.info("No table '" + this.tableName + "'. Creating one...");
                        }
                        return createTable();
                    }
                    else {
                        return rethrowAsRuntimeException(cause);
                    }
                })

So, an attempt to create a table is only after checking if it is present in the DynamoDB.

On the other hand: if you don't have your own DynamoDbAsyncClient bean, the binder configuration creates one for us:

@Bean
    @ConditionalOnMissingBean
    public DynamoDbAsyncClient dynamoDB(DynamoDbProperties properties,
            ObjectProvider<AwsClientCustomizer<DynamoDbAsyncClientBuilder>> configurer) {

Therefore your request is more confusing.

Please, elaborate or let's just close as Works as Designed!

[artembilan]

Closed as Works as Designed