KCL v2 - Checkpointing in regular intervals

[markvdlaan93]

I'm currently experiencing issues with the Kinesis Consumer Library and DynamoDB. I'm using DynamoDB for managing the checkpoints for the Kinesis consumer (like the documentation). The problem is that the checkpoints are only updated when the application is gracefully closed. I expect that the checkpoints should also be automatically updated in a regular interval (i.e. every 30 seconds or minute). Is it necessary to manually checkpoint or should the KCL take care of this?

If manually checkpointing is necessary, the documentation should be updated accordingly (https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl-java.html).

At the moment, I added a fix (from this example with v1: https://github.com/aws-samples/amazon-kinesis-learning/blob/learning-module-1/src/com/amazonaws/services/kinesis/samples/stocktrades/processor/StockTradeRecordProcessor.java) to make sure that the checkpointing happens periodically (in processRecords method after processing the batch):

private static final long CHECKPOINT_INTERVAL_MILLIS = 60000L;
private long nextCheckpointTimeInMillis;
public void processRecords(ProcessRecordsInput processRecordsInput) {
     // code for processing records like documentation
     ...........
     // Checkpoint if minute has passed
     if (System.currentTimeMillis() > nextCheckpointTimeInMillis) {
            try{
                processRecordsInput.checkpointer().checkpoint();
            } catch (InvalidStateException e){
                LOG.error("InvalidStateException thrown during checkpointing");
            } catch (ShutdownException e){
                LOG.error("ShutdownException thrown during checkpointing");
            }
            nextCheckpointTimeInMillis = System.currentTimeMillis() + CHECKPOINT_INTERVAL_MILLIS;
      }
}

[synhershko]

I believe the docs in https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl-java.html show this manual / periodic checkpointing now?

[markvdlaan93]

At time of writing, I didn't but now it does indeed. I will close this issue.

[rahulj51]

@markvdlaan93 , I see that you have closed this issue but the docs in https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl-java.html still don't show manual checkpointing.

[code4dc]

As of July 2021 the docs still don't include periodic checkpointing. This issue should be reopened.