Add exponential back-off if we get a 429

Cadair / parfive

An asyncio based parallel file downloader for Python 3.8+

https://parfive.readthedocs.io/

MIT License

50 stars 24 forks source link

Add exponential back-off if we get a 429 #130

Open Cadair opened 1 year ago

Cadair commented 1 year ago

Currently we just treat a 429 as an error and cancel the build, but backing off for a certain amount of time (without releasing the parallel slot) should mean we eventually starve out the parallel downloads until we have a number of active parallel connections the server is happy with.