search

awslabs / amazon-kinesis-client

Client library for Amazon Kinesis
Apache License 2.0
644 stars 467 forks source link

Possible for multiple applications to reuse the same workerMetricsTableName and coordinatorStateTableName #1399

Open nlm4145 opened 2 days ago

nlm4145 commented 2 days ago

Is it at all possible for multiple KCL applications to re-use the same worker metrics & coordinator state table names?

Our organization uses several hundreds of dynamoDB tables (500+), which we are replicating CRUD events to Kinesis via DynamoStreams (and subsequently read by our KCL consumers). It is already a stretch to require a unique dynamo table for each KCL application to checkpoint their progress reading the streams; adding two more per application in V3+ would absolutely be a deal breaker for our organization and seems generally and almost intentionally wasteful.

Please add a configuration option to allow for the re-use of dynamodb tables for checkpointing, worker metrics & coordinator states cross-application.

minuhong-aws commented 2 days ago

Hi @nlm4145, thanks for the input. Can you email kinesis-kcl-feedback@amazon.com for further discussion? DynamoDB Streams currently only supports KCL 1.x, so KCL 3.x cannot be used at the moment for that use case.

nlm4145 commented 2 days ago

Just to make sure we're talking about the same thing - i'm talking about this: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html

we are not using KCL to read from dynamo streams directly. either way, the point stands regarding the excessive creation of dynamo tables for each KCL application.

minuhong-aws commented 2 days ago

@nlm4145 I see. There are two ways to process CDC data from DynamoDB - 1/ use Kinesis Data Streams, 2/ use DynamoDB Streams. DynamoDB Streams currently supports KCL 1.x only, so I wanted to clarify that. And yes, this is not directly related to the issue you raised - we will review it internally and follow it up via email.