Closed mkowaliszyn-coursera closed 2 years ago
Thank you for the detailed bug report!
@mkowaliszyn-coursera - if you have a free moment and are game for an experiment, would you please try building the binary from this PR and see if it improves performance with your use case? Thanks!
Thanks @zackproser! I'll give it a try, though it may take a week or longer.
Is your feature request related to a problem? Please describe. GitHub's secondary rate limits prevent the creation of many resources in a short time frame. For example, creating PRs is capped in a period and GitHub's limits kick in.
git-xargs
will report an error and failure to create a PR when this happens with a HTTP 503 error.git-xargs
should respect the rate-limiting headers that GitHub returns and pause/sleep for that amount of time before continuing. The secondary rate limiting prevents the use ofgit-xargs
against large amounts of repos (dozens to hundreds) because of these limits. The update scripts need to add a long pause that kills any performance (i.e. 30s) and concurrency when dealing with many repos.https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits
Describe the solution you'd like It would be great if
git-xargs
respected the GitHubX-RateLimit-*
headers and automatically slept for that duration before continuing with the API requests.https://docs.github.com/en/rest/overview/resources-in-the-rest-api#secondary-rate-limits
Describe alternatives you've considered The workaround is to put a fixed pause in the update script, up to 30s to avoid being throttled and having the batch job fail.
Additional context