github: scaling for large, large repos - Githubissues

bobheadxi / timelines

🏷 Historical analysis of Git repositories and Git host activity as a service

https://bobheadxi.dev/timelines-recap

Apache License 2.0

3 stars 0 forks source link

github: scaling for large, large repos #26

Open bobheadxi opened 5 years ago

bobheadxi commented 5 years ago

At 5000 requests per hour, a repo like https://github.com/Microsoft/vscode will take more than 14 hours to sync all 70,000+ issues and PRs (at time of writing). Some things to do:

[ ] if rate limit is hit, need to put the job back on the queue, to be picked up after an hour
[x] improve how PRs are fetched (right now we list all issues, which is pretty efficient, but if an issue is a PR a separate API request is made to grab it. repos with 5000+ PRs will hit the limit fast)

also:

probably can't naively sync everything per sync - will probably have to leverage webhooks, and only ever do a full sync exactly once (#39)