search

uber-archive / hyperbahn

Service discovery and routing for large scale microservice operations
MIT License
395 stars 57 forks source link

Circuit breaker too aggressive during service restart #194

Open Raynos opened 8 years ago

Raynos commented 8 years ago

If a service gets redeployed, it will stop and start a bunch of workers.

For each worker that's restarted it will ECONNRESET the socket and kill some inflight RPCs.

The circuit breaker immediately kicks in and things there is something wrong with the service.

We should ensure that we can do a rolling restart of all instances of a service without the circuit breaker penalizing all the caller.

cc: @kriskowal @jcorbin @rf

blampe commented 8 years ago

Wouldn't per-process circuits get around this? I think we had thrown that idea around before.

Raynos commented 8 years ago

Even if we had per process circuits we still need to avoid triggering the per endpoint circuit.

We need per caller and per endpoint circuits.