search

circus-tent / circus

A Process & Socket Manager built with zmq
http://circus.readthedocs.org/
Other
1.55k stars 258 forks source link

Quit circus when watcher crashes #1103

Closed maingoh closed 5 years ago

maingoh commented 5 years ago

I am using circus to handle multiple processes inside a container, but when used with kubernetes it actually hide the crash to kubernetes and we don't know that the container is failing until we watch the logs. How complicated would it be to add an option to circus endpoint or each watcher to quit circus when the watchers crash ? So that we can let kubernetes restarting the pod.

I would be ready to implement such behavior, Thank you,

maingoh commented 5 years ago

Any advise ? Is it something you would be interested in ?

k4nar commented 5 years ago

I don't have a strong opinion on this. I'm not sure Circus is the best tool for managing several processes in a container, especially with Kubernetes. But I can see the appeal.

Maybe it could be implemented with a hook? (https://circus.readthedocs.io/en/latest/for-devs/writing-hooks/)

maingoh commented 5 years ago

Thank you, I will take a look at the hooks !

I don't know any alternative except supervisord but I guess it will be the same issue ? It is not always easy to have only one process per container.

maingoh commented 5 years ago

So I tried all hooks but unfortunately none of them allow me to check if a process crashed with a bad exit code. Would you be interested If I provide a on_failure or any other hook name ?

maingoh commented 5 years ago

I added a new type of hooks in #1104 to solve my use case. This allows to know when a process is reaped. Could you take a look ?

maingoh commented 5 years ago

Done thanks to after_reap hook