NerdWalletOSS / kinesis-python

Low level, multiprocessing based AWS Kinesis producer & consumer library
Other
118 stars 50 forks source link

Add locking and state #2

Closed borgstrom closed 7 years ago

borgstrom commented 7 years ago

This PR adds shard coordination through locking and state. Currently we support DynamoDB, but it would be trivial to support other backends if we ever required in the future.

The locking logic is:

Along with locking we now checkpoint our stream into the same DynamoDB table. The logic for checkpointing is: