Rate limit github clone calls

cbrand commented 6 months ago

Description

Fixes #139.

Add rate limiting to github clone calls, avoiding issues with github "permission denied" and EOF errors when cloning too many repositories in parallel.

TODOs

Read the Gruntwork contribution guidelines.

[x] Update the docs.
[x] Run the relevant tests successfully, including pre-commit checks.
[x] Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
[x] Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes

Added

Rate limiting Github Clone calls to 4 per default, allows being parametrized by the --max-concurrent-clones flag

Migration Guide

None needed

XavierChanth commented 4 months ago

For the reviewers, I was able to build git-xargs and used this build to clone ~80 repos mostly successfully, while the current release was only cloning 1 repo.

flinn commented 4 months ago

@brikis98 @hongil0316 What all is necessary in order to help get these changes merged/published? Asking as we've started to use git-xargs recently and without this feature we'll be very unlikely to proceed in using this tool as we're attempting to use it across ~300+ repositories.

gruntwork-io / git-xargs