search

patrobinson / gokini

A pure Golang port of the Kinesis Consumer Library
MIT License
60 stars 8 forks source link

"SequenceID" appears to be blown away in checkpoint table when lease is obtained or renewed #3

Closed robertaves closed 5 years ago

robertaves commented 5 years ago

based off quick debug, looks like seq number is blown away via the dyanamo checkpointer function GetLease . I'm experiencing on shutdown / restart that i'm going back to the beginning of my stream and replaying events because FetchCheckpoint is looking for SequenceID and it no longer exists, only the Checkpoint is present in some situations depending on timing.

checkpointer.conditionalUpdate is only marshaling the following attributes ShardID, AssignedTo, and LeaseTimeout which in turn removes the "SequenceID" attribute

AWS docs state "PutItem Creates a new item, or replaces an old item with a new item. If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item. "

Let me know you thoughts or if i misread into this.

patrobinson commented 5 years ago

Thanks @robertaves I think I've found two potential race conditions that could cause the Checkpoint to get blown away. I've created some failing tests in #4 to exhibit this behaviour and will work on a fix.