search

harlow / kinesis-consumer

Golang library for consuming Kinesis stream data
MIT License
263 stars 88 forks source link

Detect checkpoint seqnum pointing to expired data #126

Open jtackaberry opened 4 years ago

jtackaberry commented 4 years ago

Implementation for #125 for consideration/discussion.

CI will fail but I can add a mock for DescribeStreamSummaryWithContext if you think this is worthy of merging.

This also adds an extra ticker fastTicker which is based on the 5TPS rate limit, and which is used for those cases where we want a fast return back to Kinesis. I can amend #122 to use this as well, if/when the time comes. Alternatively, to squeeze more throughput, we could use the behavior from #122 (continue immediately) and only start backing off if we hit ErrCodeProvisionedThroughputExceededException.

lepek commented 1 year ago

Hi! What's the status of this? It hasn't been merge right? I am running into the same issue