GCP rate limit - Githubissues

google-github-actions / auth

A GitHub Action for authenticating to Google Cloud.

https://cloud.google.com/iam

Apache License 2.0

966 stars 196 forks source link

GCP rate limit #427

Closed GregoireW closed 4 months ago

GregoireW commented 4 months ago

TL;DR

More like a discussion than an issue, but I ask here. I got a "[Security Token Service] The request was throttled due to rate limit. Please retry after a few seconds." error using this auth action.

My question would be:

what is the rate limit ? (seems like 100 call per second but ... I'm not sure it is on a documentation somewhere, so if someone knows... ) I guess it is a global rate limit and we cannot modify it per project basis ?
In my case, there is a spike in the number of call on the pool, and I would like to drill down what the culprit. I think to remember a discussion about workload identity logs, but I don't find it and I don't really remember the conclusion (at one point it was question that iam api are not project bound so it is hard to get logs, but I really don't remember...) If someone can help ...

Thank you.

Detailed design

No response

Additional information

No response

github-actions[bot] commented 4 months ago

Hi there @GregoireW :wave:!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

sethvargo commented 4 months ago

https://cloud.google.com/iam/quotas screenshot-20240716-qhZf1phC@2x

GregoireW commented 4 months ago

..... Why did I never find what I need when I need ? ..... Thank you !!

And to sump up ... yep, in my situation exchange token requests is pretty high sometime

and as it is a spike from time to time, the api is really low and do not show if you don't watch exactly when this occurs. Anyway I found the service, asked to increase the quota and that all. Again, Thank you.

sethvargo commented 4 months ago

@GregoireW what are you doing that results in so many sts requests?

GregoireW commented 4 months ago

@sethvargo If I could analyses the request ( the id_token subject for instance) it would be easy to answer :D but basically, we don't want json file credentials, we have a lot of repository, and so a lot of github action are executed .

In this particular pool, we have request to get some artifact from artifact registry and we also have a gcs to receive build report/flle ( detailled test report, front end test may post video // report with screenshot when error .... ) exposed internally to the developpers ...
To make this worst, There is a particular (legacy) repository that have multiple workflows when a change is done on a PR (don't ask why, I will start to cry ) and that have a lot of open PR (there is a lot of open feature).... When a sync is done on those PR, 100s of job are run... My guess is this is the reason ...But even if I got some spike on workflow job it is hard to link those with the spike on the GCP api.

sethvargo commented 4 months ago

Can you limit the concurrency of these jobs?

GregoireW commented 4 months ago

As long as I can do without this I will I guess. I asked to increase the quota, the validation has go through so now I should not be bothered by that ... As the repo should decrease in activity (well it is not my application, but the path to migrate to another solution exist) I hope in the future this would not happens for this reason. On the other hand, if this happens because a solution is used on all our repo, then it is a good thing