search

twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Fetch shard IDs from Kinesis if DynamoDB cache is empty #22

Closed dcelasun closed 5 years ago

dcelasun commented 5 years ago

On startup, if the metadata table is being created for the first time, loadShardIDsFromDynamo() call in refreshShards() will always return an empty shard list.

This causes the consumer to wait until the next leader election to start receiving events, because that's the only time loadShardIDsFromKinesis() is called. By default, this happens once a minute, which means the consumer basically sleeps for 1 minute on first run.

This commit fixes that by adding a first run flag and skipping the cache if it's true.

Fixes #17.

garethlewin commented 5 years ago

(Merged master into this branch to get the update Travis CI changes.)

dcelasun commented 5 years ago

This is ready now, but #21 should be merged first.

garethlewin commented 5 years ago

Sorry for delays. I have updated master to build correctly if you want to run tests again.

garethlewin commented 5 years ago

If not I feel confident enough to merge this in.

dcelasun commented 5 years ago

@GarethLewin all tests are green, this can be merged.