search

urakozz / go-dynamodb-stream-subscriber

Go channel for streaming Dynamodb Updates
MIT License
32 stars 19 forks source link

checkpointing for streams #4

Open abhineshada opened 5 years ago

abhineshada commented 5 years ago
urakozz commented 5 years ago

What the hell is checkpointing? I didn't use DynamoDB for some years, is is new term there?

How to handle missing/duplicated data?

However you like, this library only wraps complexity of iterating over overengineered dynamo streams

abhineshada commented 5 years ago

By checkpoint, I mean tracking the status of shards/data processed.

Refer: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html https://www.infoq.com/articles/demystifying-dynamodb-streams

KCL adapter for dynamoDB streams has checkpoint feature (only available in Java): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html

The library is really well written. I had few doubts regarding how data accuracy can be maintained using go SDK.