search

twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Allow starting from LATEST position #16

Closed dcelasun closed 5 years ago

dcelasun commented 5 years ago

This implements support for ShardIteratorTypeLatest. By setting SequenceNumber in DynamoDB to LATEST, consumers can start with the latest records in the stream.

dcelasun commented 5 years ago

@GarethLewin any chance you or someone else can take a look at this?

garethlewin commented 5 years ago

(Merged master into this branch to get the update Travis CI changes.)

garethlewin commented 5 years ago

@dcelasun So what is the workflow you propose for this, manually modifying the entries in the table?

I have considered adding a config flag to start from latest, but I'm not sure if that achieves your goal since maybe you want to do this only for specific shards?

dcelasun commented 5 years ago

manually modifying the entries in the table?

Yes, which also works for TRIM_HORIZON.

maybe you want to do this only for specific shards?

Indeed, it's very useful for debugging when you are only interested in a single shard and you don't care about past events.