twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Refactor shard_consumer and handle expired iterators #50

Open FugiTech opened 4 years ago

FugiTech commented 4 years ago

This moves fetching records into a goroutine to try and abstract the logic in preparation for enhanced fanout. It also moves the commit loop into a goroutine to try and simplify consume() a bit. Line 248 also adds handling for expired iterators.

https://github.com/Fugiman/kinsumer/pull/1 shows the enhanced fanout changes which rely on this PR. Let me know if you'd prefer to review them together.