Implement buffered channel for pull requests

zackproser commented 2 years ago

Description

Fixes #59

This PR addresses #59 by implementing several new features and flags to help us respect GitHub's API rate limits when using git-xargs:

Distinct processing channels for expensive and non-expensive work
Automatic, and configurable, pull request retries when rate limited
Automatic, and configurable, backoff when rate limiting is detected
Making the pause between pull requests configurable

In addition, three new flags to assist with configuring your git-xargs jobs so that they do not trip GitHub rate limits:

--seconds-between-prs The number of seconds to wait between pull requests when serially opening them at the end of a run. In local testing, setting this value to 12 seconds resolves the secondary rate limiting issues seen reliably on the same test data set prior to making these changes.
--seconds-to-wait-when-ratelmited The number of seconds to wait when git-xargs detects it has been rate limited
--max-pr-retries The number of times to retry a failed pull request before abandoning it

TODOs

Please ensure all of these TODOs are completed before asking for a review.

[x] Ensure the branch is named correctly with the issue number. e.g: feature/new-vpc-endpoints-955 or bug/missing-count-param-434.
[x] Update the docs.
[x] Keep the changes backward compatible where possible.
[x] Run the pre-commit checks successfully.
[x] Run the relevant tests successfully.

Related Issues

rhoboat commented 2 years ago

Amazing that you have time to do this as well as everything else you're doing. I won't be able to test this locally, as I currently have no discretionary time, but the code looks good to me, as far as readability, naming, logic, etc.

zackproser commented 2 years ago

Thanks for the thorough review, @rhoboat! I'll get your changes in.

rhoboat commented 2 years ago

GitHub seems to have completely lost my review comments. I said something like:

LGTM! The addition of PRChan and processing using throttling makes a lot of sense!

One suggestion: it took me a little while to understand the overall logic flow. Maybe it makes sense to make a line diagram of how the processing works, where the delays, retries, and concurrency happen. Could help future-you too.

rhoboat commented 2 years ago

I left a review comment that again disappeared (thanks GH, ❤️). I said:

I left nits and some suggestions for code organization, which are up to you. This looks great! I'd love to see this merged.

zackproser commented 2 years ago

Thanks so much for the reviews! Going to merge this in now.

mkowaliszyn-coursera commented 2 years ago

Thanks so much for this!

gruntwork-io / git-xargs