Checkpoints processed records (handled remotely so tolerant of a machine disappearing)
Coordinates shard associations with other workers
Balances shard-worker associations
The way KCL implements this is through a dynamodb table where it keeps track of the current owner/checkpoint and other info in a single row per shard per application
It would be nice to have a haskell library that matches the feature set of the KCL.
http://docs.aws.amazon.com/kinesis/latest/dev/developing-consumer-apps-with-kcl.html
Some notable features:
The way KCL implements this is through a dynamodb table where it keeps track of the current owner/checkpoint and other info in a single row per shard per application