search

harlow / kinesis-consumer

Golang library for consuming Kinesis stream data
MIT License
264 stars 90 forks source link

The timestampInMillis parameter cannot be greater than the currentTimestampInMillis #116

Closed adamcypher closed 3 weeks ago

adamcypher commented 4 years ago

using:

consumer.WithShardIteratorType("AT_TIMESTAMP"),
consumer.WithTimestamp(initTimestamp),

I've had to increase our grace period from 30 seconds 60 seconds to account for clock skew between our environments and Kinesis.

initTimestamp := time.Now().Add(-time.Second * 60)

It would be cool if the library would detect this error and readjust the timestamp by parsing the differential of the values returned in the error message, and retry. I'll consider making a PR if no one else picks this up.

The error message looks like this:

shard shardId-000000000000 error: get shard iterator error: InvalidArgumentException: The timestampInMillis parameter cannot be greater than the currentTimestampInMillis. timestampInMillis: 1593030726246, currentTimestampInMillis: 1593030724560

We only using AT_TIMESTAMP for tests and local development, production uses WithStore, so it's a minor annoyance.

harlow commented 4 years ago

It would be cool if the library would detect this error and readjust the timestamp by parsing the differential of the values returned in the error message, and retry

Cool idea! if you have time to make a PR that would be amazing ❤️