search

awolden / brakes

Hystrix compliant Node.js Circuit Breaker Library
MIT License
300 stars 35 forks source link

Edge case where a circuit can be opened but report failure stats below the circuit breaking threshold #10

Closed seanohollaren closed 8 years ago

seanohollaren commented 8 years ago

Found an edge case where the circuit can be opened, but brakes is showing request stats that indicate that it shouldn't have opened (because failure percentage is below threshold).

I think what might be happening is:

  1. With several requests pending, one comes back as a failure.
  2. Brakes sees this puts the server over the user-specified failure threshold (e.g. 50%) and opens the circuit, reporting a failure rate of 51%.
  3. The other pending requests return successfully after the circuit has opened, which alters the request stats (bringing failure percentage down to 48%).
  4. Brakes rejects all new requests to the server, but reports that it's doing so because the failure rate (48%) is higher than the failure threshold (50%).

This isn't a case of brakes doing the wrong thing. It's just a rare case (which will probably only happen at low numbers of requests) where the way brakes reports things could be confusing.

awolden commented 8 years ago

This issue has been fixed in release 2.0