gbataille
commented
7 years ago Hey,
so I found the issue.
Actually rainbow-saddle assumes strongly that the gunicorn second arbiter will start and produce the .pid.2 file. If it does not, for whatever reason
- rainbow-saddle is stuck in an infinite loop waiting for the file.
- gunicorn still runs the old arbiter
What is very problematic here is that it's all silent. The rainbow-saddle process is still up, gunicorn is still up too. The only thing is a stack trace in the gunicorn error log, but since the old master still runs, it actually gets lost in the uninterrupted activity
Hey,
have you ever had the case where the rainbow-saddle process is up. The gunicorn arbiter and workers are up. The pid file is as expected
BUT sending a HUP to rainbow-saddle does absolutely nothing (no log trace, no nothing)?
I don't know what to do or how to investigate short of adding traces in rainbow-saddle code (and still I'm not sure how to go about the issue)
To be clear, I can't reproduce it at will, it just happens from time to time on our server and I need to do a manual restart (getting the server out of the load balancer and stuff like that first)