edgi-govdata-archiving / wayback

A Python API to the Internet Archive Wayback Machine
https://wayback.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
61 stars 12 forks source link

Stop retrying after rate limit errors #146

Closed Mr0grog closed 9 months ago

Mr0grog commented 9 months ago

We used to retry requests that got a response with a 429 status code (indicating that you've hit a rate limit on the server) automatically, but we've resolved that this is no longer a good approach.

See the discussion in https://github.com/edgi-govdata-archiving/wayback/issues/137#issuecomment-1828873225 for more detail, but the short version is that our previous retry behavior was specifically geared to a workflow EDGI used, and which had custom rate limits and other server behavior that most users won't have, and also helped to work around a bunch of deficiencies in our client-side rate limiting that were solved in the last release (but they do need a bit more work in this release, too). It turns out this led to some users getting blocked, and it's better to leave handling this situation to user code.