In the case where the pool experiences a failure on every server (especially apparent when pool size is 1), the pool does not back-off on a very recently failed request, causing a stampede effect on that server. Instead, we should be tracking the health of each server in the pool and exponentially backing off on those requests. There may be different classes of requests which require different backoff handling, but that is outside the scope of this feature.
For now, any failure which would normally trigger a rotation, should be recorded and considered when making a request at a "recently-failed" server.
[x] A failure should place a timer on that server's "availability" which decreases to zero.
[x] Recently-failed is defined as any server with an active "failure" timer running.
[x] Requests will block (up to their context's timeout) until an available server returns as ready to complete a request and consumes the request from the stack of waiting requests.
[x] Replay TransactionListening and AddressListen calls after rotation
In the case where the pool experiences a failure on every server (especially apparent when pool size is 1), the pool does not back-off on a very recently failed request, causing a stampede effect on that server. Instead, we should be tracking the health of each server in the pool and exponentially backing off on those requests. There may be different classes of requests which require different backoff handling, but that is outside the scope of this feature.
For now, any failure which would normally trigger a rotation, should be recorded and considered when making a request at a "recently-failed" server.