search

awslabs / dynamodb-streams-kinesis-adapter

The Amazon DynamoDB Streams Adapter implements the Amazon Kinesis interface so that your application can use KCL to consume and process data from a DynamoDB stream.
Apache License 2.0
97 stars 37 forks source link

Only one worker get records when multiple Workers on same stream #35

Closed subash89 closed 4 years ago

subash89 commented 4 years ago

We have a listener worker connecting to a dynamodb table stream to read item insert events. All works fine with 1.5.1 version. But the moment I start another JVM with the same listener worker, only the first one keep getting records/events. In the secondly started listener, I can see following log. So my question is, given this works with one kinesis adaptor client and second consumer not receiving events, how can we ensure each KCL client pointing to same stream get a copy the event. Looks like only one get it ?

INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - No activities assigned INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Sleeping ...

aggarwal commented 4 years ago

By adding another worker, you could be trying to achieve one of two outcomes: (1) increase the number of workers in a single application, or (2) have multiple applications processing the same stream. What is the intent of the second worker?

Stream shards without an ending sequence number are "active", meaning they might have more data written into them by the base DynamoDB table. How many such active stream shards are in your stream? -- It's possible that there aren't enough shards for two workers to work on them.