search

WW-Digital / reactive-kinesis

A Scala & Akka based implementation for working with Amazon Kinesis Streams
Apache License 2.0
34 stars 13 forks source link

Investigate & implement 2 Phase checkpoints to reduce duplication on failover #43

Open markglh opened 6 years ago

markglh commented 6 years ago

Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint. Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.

See: https://github.com/awslabs/amazon-kinesis-client/pull/188

This passes the pendingCheckpoint to the new RecordProcessor upon failover, allowing it to continue at that record if the actual checkpoint failed.