"gh pr create" failed with "GraphQL: was submitted too quickly (createPullRequest)"

[mikeharder]

> gh pr create ... 
pull request create failed: GraphQL: was submitted too quickly (createPullRequest)

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3215735&view=logs&j=b5398758-6341-5a90-e082-88d59643ae40&t=43f26e3f-f7ac-5721-717f-66d736603ab9&l=25

Source: https://github.com/Azure/azure-sdk-tools/blob/5af9b9ad82b0bd8bd1b9463ae2254c22ea976d80/eng/pipelines/templates/steps/sync-repo-merge-branch.yml#L77

Backgrouind

was submitted too quickly is an intentional restriction on the platform to combat abuse by automated actors. Unfortunately, this rate limit is internal (since it can vary dynamically) and it also affects people who just want to legitimately create a lot of objects at once. The solution is to scan for this error message in your programs and retry creation after a delay of some minutes or up to an hour.

https://github.com/cli/cli/issues/4801#issuecomment-1029207971

[benbp]

@mikeharder if you limit the requests to one per second that's typically the easiest way to avoid the secondary rate limit, assuming no other throttling conditions are met.

[weshaggard]

We are currently already only doing this every few seconds assuming it is just the mirroring pipeline that is causing the throttling. We can try and insert a delay or try to handle the error. Another issue that should be fixed is we need to correctly handle this error and fail the step, or at least show as yellow so that we know something went wrong without needing to dig into all the logs manually.

[mikeharder]

@mikeharder if you limit the requests to one per second that's typically the easiest way to avoid the secondary rate limit, assuming no other throttling conditions are met.

@benbp:

I've reversed-engineered the limits (as of right now) to be 20 issues per minute and 150 per hour

https://github.com/cli/cli/issues/4801#issuecomment-1430651377

In this case, I suspect we were hitting the "150 per hour" limit, since I got the same error when I retried shortly after the first error, but it worked when I tried over an hour later.

[weshaggard]

For reference there is some info about these limits at https://docs.github.com/en/rest/overview/rate-limits-for-the-rest-api?apiVersion=2022-11-28#calculating-points-for-the-secondary-rate-limit. It seems that mutation counts as "5" towards the limit.