xh3b4sd
commented
1 year ago I wrote https://github.com/xh3b4sd/budget a while ago and used it successfully in various projects. In theory you want to consider error budgets based on system health and response capacity. Depending on scale you can draw multiple lines of defence and implement more sophisticated protocol level schemes for probing and resuming. To start small constant or exponential time based backoff implementations are common. Happy to chat about this.
Right now, at least labelmaker will try to re-connect to the upstream server very aggressively (hundreds/thousands of times per second) if it receives a valid error. It should probably do an exponential backoff thing.
I vaguely remember the BGS doing some backoff when dialing PDS instances, but I could be confused.
cc: @whyrusleeping