search

mergestat / mergestat-lite

Query git repositories with SQL. Generate reports, perform status checks, analyze codebases. 🔍 📊
https://mergestat.com/
MIT License
3.47k stars 105 forks source link

Retries on GitHub API tables? #258

Open patrickdevivo opened 2 years ago

patrickdevivo commented 2 years ago

Currently, it's pretty easy to hit a GitHub rate limit when running queries that use the GitHub API tables. We could consider implementing a back-off-retry strategy to pause execution when we encounter one, wait an appropriate amount of time, and retry/continue

klauern commented 2 years ago

Seconded. I have hit a lot of rate limits and there's definitely opportunity here to know what the existing rate limit is and work with it. I don't think it helps that some queries using the v4 API will eat up a lot more of your quota than an equivalent v3 call. We have a small utility that gathers all the pull requests for a repo, and if you have a highly active one, this seems ideal to leveraging mergestat for. Unfortunately, I haven't been able to find a magic combination on the GITHUB_PER_PAGE and GITHUB_RATE_LIMIT variables that doesn't eventually end up in a timeout. I can get further, but never all the results, and it takes a LOT longer than an equivalent REST v3call to the Pulls API.