Manually update checkpoint

[Sytten]

Hi! Is it possible to manually update the checkpoint manually? My use case is that the processing of a record can fail temporarily and I would like to be able to replay the same record again. Thanks

[garethlewin]

There is currently no way. Because of kinsumer's queue setting the checkpoint backwards in time would cause duplicate reads, are you ok with that?

I am not sure how we would expose this though.

[Sytten]

In my use case, this is fine because that means the consumer failed to process the message so I want it to try again by reading it again. Though with the current architecture of the lib it might be hard to do. Especially since the lease can jump from one consumer to the other.

[soenkehahn]

We're also looking for a way to do this. We would like to

• retrieve a record from kinsumer,
• process it somehow (unrelated to kinesis),
• if there's errors during this processing:
  • Don't update the checkpoint, since we want to retry processing it.
  • Destroy the currently used kinsumer,
  • Spin up a new one and restart the consumption of records.

Right now, as we understand, if we either call Next or Stop, the library will think we successfully processed the event, update the checkpoint and not re-send the record that we failed to process. Is that understanding correct?

One workaround seems to be to just not call Next nor Stop and let the client die (and probably spin up a new one). After 5 minutes (by default) kinsumer will send the failed record to another client. That doesn't seem ideal, since we have to wait for 5 minutes.

It'd be nice if there was another function, e.g. Fail, that we could call on a Kinsumer to let it now that we

1. don't want it to update the checkpoint and
2. won't be using that Kinsumer anymore.

I'm a bit new to kinesis, so my understanding might be wrong, sorry.