harlow / kinesis-consumer

Golang library for consuming Kinesis stream data
MIT License
269 stars 90 forks source link

Expired shard iterator is not retried #166

Open srebhan opened 1 week ago

srebhan commented 1 week ago

When consuming data and disconnecting the consumer for a longer time, the persisted shard iterator expires. However, this should not lead to an error but rather the shard-iterator should be renewed. Instead we see

... shard shardId-xxxxxxxxxxxx error: get records error: operation error Kinesis: GetRecords, https response error StatusCode: 400, RequestID: aaaaaaaa-bbbb-cccc-ddddd-eeeeeeeeeeee, ExpiredIteratorException: Iterator expired. The iterator was created at time Thu Nov 21 21:38:33 UTC 2024 while right now it is Thu Nov 21 21:54:14 UTC 2024 which is further in the future than the tolerated delay of 300000 milliseconds.

To me this hints for an ExpiredIteratorException, however, the isRetriable function obviously doesn't match the above error and Scan exits with an error...

geronimo-iia commented 21 hours ago

Same behavior with ProvisionedThroughputExceededException !