Improving throughput of API requests

[julianharty]

Context

GitHub API has a rate limit where they reject incoming requests if they occur too frequently with a HTTP 403 status code. Currently the code has various hard coded time.sleep(...)'s to cope when these occur and also a loop that attempts a request up to 10 times.

I reckon we can improve the throughput and also implement a simple backoff algorithm based on the number of attempts the code has performed for a particular request. We can iteratively improve the code if any immediate changes aren't sufficient to prevent the requests being rate limited repeatedly. Conversely we don't want the code to pause excessively as doing so increases the overall runtime of the script.

[julianharty]

Notes:

• GitHub limits the throughput to 10 requests per minute: https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#:~:text=This%20endpoint%20requires%20you%20to%20authenticate%20and%20limits%20you%20to%2010%20requests%20per%20minute.
• GitHub provides live results about rate limits via an API request https://docs.github.com/en/rest/rate-limit/rate-limit?apiVersion=2022-11-28
• GitHub provides information when rejecting requests that they recommend we abide by https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api?apiVersion=2022-11-28#handle-rate-limit-errors-appropriately and https://docs.github.com/en/rest/using-the-rest-api/troubleshooting-the-rest-api?apiVersion=2022-11-28#rate-limit-errors