search

twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Startup takes 1 minute when DynamoDB shard cache doesn't exist #17

Closed dcelasun closed 5 years ago

dcelasun commented 5 years ago

When starting a Kinsumer for the very first time, it takes quite a bit of waiting to start receiving events because Kinsumer doesn't read the shard IDs from Kinesis. It only checks the cache in DynamoDB, which is naturally empty since it was just created.

The only place where DescribeStream is called to get shard IDs is in performLeaderActions which by default happens every 1 minute. This means that when a Kinsumer is started for the first time, it basically sleeps for 60 seconds.

Kinsumer should have a flag, firstRun, that skips the DynamoDB cache and directly call loadShardIDsFromKinesis if true.

garethlewin commented 5 years ago

Hello @dcelasun sorry I have not responded to this and your other issues/PRs, I am currently traveling and will address them when I get back. Thanks a lot for your efforts.

dcelasun commented 5 years ago

Hey @GarethLewin, now that I know the repo is alive, there is no rush :)