GitHub API has a rate limit where they reject incoming requests if they occur too frequently with a HTTP 403 status code. Currently the code has various hard coded time.sleep(...)'s to cope when these occur and also a loop that attempts a request up to 10 times.
I reckon we can improve the throughput and also implement a simple backoff algorithm based on the number of attempts the code has performed for a particular request. We can iteratively improve the code if any immediate changes aren't sufficient to prevent the requests being rate limited repeatedly. Conversely we don't want the code to pause excessively as doing so increases the overall runtime of the script.
Context
GitHub API has a rate limit where they reject incoming requests if they occur too frequently with a HTTP 403 status code. Currently the code has various hard coded
time.sleep(...)
's to cope when these occur and also a loop that attempts a request up to 10 times.I reckon we can improve the throughput and also implement a simple backoff algorithm based on the number of attempts the code has performed for a particular request. We can iteratively improve the code if any immediate changes aren't sufficient to prevent the requests being rate limited repeatedly. Conversely we don't want the code to pause excessively as doing so increases the overall runtime of the script.