Add worker auto-respawning with max tries

LearnBoost / up

Zero-downtime reloads and requests load balancer based on distribute.

540 stars 73 forks source link

Add worker auto-respawning with max tries #2

Open rauchg opened 12 years ago

rauchg commented 12 years ago

Option to ensure that if a worker dies it tries to respawn immediately ?
Option to ensure number of active workers always stays at numWorkers or above ?
Option to ensure that there's always at least one worker alive and otherwise respawn ?
When respawning, if worker fails to listen within certain timeout, retry ?
Have a retry threshold to avoid overload ?
When respawning, if worker dies shortly after listening, consider it an unsuccessful respawn and try ? This means it would still be captured by the retry threshold and you can't get into an endless cycle of considering the worker spawned, then dying shortly after listening, then triggering reload, ad infinitum.

malkomalko commented 12 years ago

Hi Guillermo,

First of all... really loving up. Right now when a worker dies there is no respawn.. is that correct? Is there any current strategy at ensuring workers stay alive? Maybe listen for a worker failure and send a kill -s SIGUSR2 to the master pid?

rauchg commented 12 years ago

Right now I have nothing for worker reloading I believe. We can start with the simplest implementation, but it quickly can get out of control if you don't cover many of the things I mention in this ticket. eg: crazy recursive/cyclic reloads that bring the server down because of faulty code.

rauchg commented 12 years ago

At the same time, maybe that problem needs to be solved at a different layer of abstraction.

arohter commented 12 years ago

See my comments in https://github.com/LearnBoost/up/issues/21

kevireilly commented 11 years ago

It seems this has been integrated and documented, but has not yet been translated to the npm repository. Just a heads up.

cmawhorter commented 11 years ago

I think merging #60 will close this out.

Option to ensure that if a worker dies it tries to respawn immediately ? Already Fixed
Option to ensure number of active workers always stays at numWorkers or above ? Already Fixed
Option to ensure that there's always at least one worker alive and otherwise respawn ? _Fixed by #60 _
When respawning, if worker fails to listen within certain timeout, retry ? _Fixed by #60 _
Have a retry threshold to avoid overload ? _Fixed by #60 _
When respawning, if worker dies shortly after listening, consider it an unsuccessful respawn and try ? This means it would still be captured by the retry threshold and you can't get into an endless cycle of considering the worker spawned, then dying shortly after listening, then triggering reload, ad infinitum. _Fixed by #60 _