actions / runner

The Runner for GitHub Actions :rocket:
https://github.com/features/actions
MIT License
4.64k stars 892 forks source link

Backoff to avoid excessive retries to Run Service in a duration #3354

Closed ericsciple closed 6 days ago

ericsciple commented 1 week ago

Helps avoid excessive calls to Run Service when encountering non-retriable errors from /acquirejob. Normally we rely on the HTTP clients to back off between retry attempts. However, acquiring a job involves calls to both Run Serivce and Broker. And Run Service and Broker communicate with each other in an async fashion.

When Run Service encounters a non-retriable error, it sends an async message to Broker. The runner will, however, immediately call Broker to get the next message. If the async event from Run Service to Broker has not yet been processed, the next message from Broker may be the same job message.

The error throttler helps us back off when encountering successive, non-retriable errors from /acquirejob.