We discovered a bug in the circuit breaker logic that updates the opened_at value for the storage state as new processes spin up and initialize that state. The consequence of this is that we can inadvertently extend the timeout window for an open circuit indefinitely if new processes spin up more frequently than the timeout configured.
This fix moves the setting of the opened_at time to CircuitBreaker.open() so that it gets set only when the breaker opens and not when the CircuitBreakerState is instantiated (which can happen without an attempt to call the function wrapped).
danielfm
commented
5 years ago
We discovered a bug in the circuit breaker logic that updates the
opened_atvalue for the storage state as new processes spin up and initialize that state. The consequence of this is that we can inadvertently extend the timeout window for an open circuit indefinitely if new processes spin up more frequently than the timeout configured.This fix moves the setting of the
opened_attime toCircuitBreaker.open()so that it gets set only when the breaker opens and not when theCircuitBreakerStateis instantiated (which can happen without an attempt to call the function wrapped).