exponential backoff

[shriphani]

A more diverse crawl policy.

[shriphani]

Might need to remove the implicit policy from the codebase and store it somewhere.

[dhruvbhatia]

Hey @shriphani, have you seen the https://github.com/joegallo/robert-bruce library? I quite like it as it supports both linear and exponential retries/ and has hooks for getting the retry count and choosing to backoff right out of the box.

May be a way to simply implement this rather than rewriting logic if you're happy to rely on an existing lib?

[shriphani]

this is beautiful - thanks for the link.