search

whale2 / async-kinesis-client

Python Kinesis Client library utilising asyncio
MIT License
13 stars 4 forks source link

Support passing a limit to get_records #8

Closed jhorman closed 4 years ago

jhorman commented 5 years ago

The default limit I think is pretty high. 10MB of data. I would like to be able to fetch less records, and checkpoint more often. That way if there is a failure in my record processing I can restart further into the stream.

whale2 commented 5 years ago

There's a setting for how many records to checkpoint after -

client.set_checkpoint_interval(n)

does it help?