search

NerdWalletOSS / kinesis-python

Low level, multiprocessing based AWS Kinesis producer & consumer library
Other
118 stars 50 forks source link

ShardReader retry does not back-off #31

Open inventhouse opened 2 years ago

inventhouse commented 2 years ago

This code is very helpful for understanding how to work with Kinesis Data Streams - thank you!

While sifting through it, I noticed that retries, which controls the ShardReader's back-off is never actually incremented as far as I can tell; this line adds 1 but never saves the incremented value:

https://github.com/NerdWalletOSS/kinesis-python/blob/2bf355173de05a38f685df0f5a6000cff9a2076a/src/kinesis/consumer.py#L54

This should work:

    self.retries += 1
    log.debug("Retrying get_records (#%d %ds): %s", self.retries, loop_status, exc)
inventhouse commented 2 years ago

While I'm here, this line: https://github.com/NerdWalletOSS/kinesis-python/blob/2bf355173de05a38f685df0f5a6000cff9a2076a/src/kinesis/consumer.py#L52

Should be (self.retries * 2) or 1 if the behavior is to (mostly) match the comment.