davidmurdoch
commented
9 months ago We should probably use exponential backoff + jitter + circuit breaker, and also augment it with an AbortController so we (or a user action) can cancel requests.
defining terms for those that don't know:
jitter here means we add a random value to the retry delay. This reduces load on the upstream service by spreading out MM retry requests from all happening at the exact same time. Useful if the upstream service is overwhelmed. The jitter helps spread retry requests over a longer period.
exponential backoff: the first retry can be after 1 second, second retry is in 2 seconds, third is in 4 seconds, etc. (up to a limit)
circuit breaker: once we reach the maximum backoff limit we set the circuit to an "open" state for a set period of time. This will cause any new requests to immediately fail without being tried. After a short period of time the circuit should be reset to "half open", at which point the next request will be tried once in order to determine if the service has been restored; if it works the circuit is reset to "closed" (which somewhat confusingly means it's working normally).
AbortController: https://developer.mozilla.org/en-US/docs/Web/API/AbortController
When we encounter a "retriable" error, we should use an exponential backoff to increase the chances of the request succeeding and decrease the load upon the RPC endpoint.