search

cncf / clowarden

CLOWarden is a tool that manages access to resources across multiple services
https://clowarden.io
Apache License 2.0
39 stars 5 forks source link

GitHub API rate limit #61

Open tegioz opened 1 year ago

tegioz commented 1 year ago

With the actual number of repositories and teams, a validation job takes ~20 API requests, whereas a reconciliation one takes ~500 requests. The installation access token we'll be using for the CNCF org (enterprise cloud) has a rate limit of 15000 requests per hour, which means that at the moment we should be be able to handle a bit less than 30 pull requests per hour on the CNCF people repo that modify the config.yaml permissions file.

We have some ideas to both reduce the API usage and improve how the situation is handled should the rate limit be reached, but in the meantime we wanted to bring awareness about this limitation (although we don't expect it to be a problem in the short term).

/cc @jeefy @RobertKielty

jeefy commented 8 months ago

Could using something like https://github.com/kubernetes/test-infra/tree/master/ghproxy (or a new implementation of it) be helpful?

tegioz commented 8 months ago

Thanks @jeefy! I didn't know about ghproxy/ghcahe, added to the toolbox 😉

One of the ideas I mentioned we had to reduce API usage would be similar to that: the Rust GitHub library we use in CLOWarden (octorust) includes a feature named httpcache that does something similar transparently but within the same process (we just run a single instance of the CLOWarden server).

We still need to test this thoroughly to make sure everything works as expected, but we'll keep this in mind. Thanks again 🙂