wjdp / htmltest

:white_check_mark: Test generated HTML for problems
MIT License
323 stars 54 forks source link

Limit per-host concurrency and add retries #217

Open theunrepentantgeek opened 1 year ago

theunrepentantgeek commented 1 year ago

Limits the number of concurrent requests to a single host in order to reduce (hopefully to eliminate) errors where the server objects. Also adds automatic retries (with exponential backoff) to further mitigate transient errors.

Partial mitigation of #152. Closes #103 Closes #25

The introduction of retries has required update of the VCR recordings; the actual changes are fairly limited in scope - I deliberately chose retryablehttp because it was a near seamless drop in replacement.

~Marked as draft because too many commits have been pulled in by Github.~ Fixed by rebasing this PR. Might have conflicts if #215 is merged first; I'll remedy those if that happens.