This adds a bunch of little tweaks related to the recent rate limit work. It should be more-or-less ready for release after this.
The main thing is adding a 60-second delay for 429 (rate limit) responses. Looking back at the docstring for the RateLimitError class, I think I might have originally intended to always raise instead of retry these, but git history shows it never actually worked that way. I want to get a patch release out with the new retry timings and logs, but patch release doesn’t seem like a good time change whether a response gets retried, so I’ve just made a longer delay here for now.
Then some other assorted bits and bobs:
Downgrades the log level when we catch and retry exceptions (used to be WARN, now just INFO). The whole point of retry is to make this situations not special, so I don’t think it’s actually worth a warning.
Actually handle all forms of Retry-After header, and move logic to _utils. We didn’t previously handle the date form of the header, just the integer form that indicated a number of seconds. Now we handle both. I also moved the logic out of the exceptions module since it was too much special logic (arguably this was already the case even before I improved the parsing).
Fix the total_time calculation for WaybackRetryError. I’m not sure what I was thinking with the previous code, which was… not even sort-of correct.
Add read_and_close() everywhere that we raise an exception. We did it in some spots, but had missed others.
This adds a bunch of little tweaks related to the recent rate limit work. It should be more-or-less ready for release after this.
The main thing is adding a 60-second delay for 429 (rate limit) responses. Looking back at the docstring for the
RateLimitErrorclass, I think I might have originally intended to always raise instead of retry these, but git history shows it never actually worked that way. I want to get a patch release out with the new retry timings and logs, but patch release doesn’t seem like a good time change whether a response gets retried, so I’ve just made a longer delay here for now.Then some other assorted bits and bobs:
Downgrades the log level when we catch and retry exceptions (used to be WARN, now just INFO). The whole point of retry is to make this situations not special, so I don’t think it’s actually worth a warning.
Actually handle all forms of
Retry-Afterheader, and move logic to_utils. We didn’t previously handle the date form of the header, just the integer form that indicated a number of seconds. Now we handle both. I also moved the logic out of theexceptionsmodule since it was too much special logic (arguably this was already the case even before I improved the parsing).Fix the
total_timecalculation forWaybackRetryError. I’m not sure what I was thinking with the previous code, which was… not even sort-of correct.Add
read_and_close()everywhere that we raise an exception. We did it in some spots, but had missed others.