Supervisor / supervisor

Supervisor process control system for Unix (supervisord)
http://supervisord.org
Other
8.46k stars 1.24k forks source link

startretries limit not working #1285

Open enumag opened 5 years ago

enumag commented 5 years ago

I want supervisor to try restarting my services a few times before giving up. From the documentation it seems the right configuration for that is this:

autorestart = true
startretries = 3

But when I run supervisor with this configuration and the commands fails and exists with exitcode 1, supervisor always tries to restart it, never giving up. ps keeps showing the process as running but always with a different PID. Same goes for supervisorctl -i, it always shows the process as running but always with different pid and zero uptime:

/usr/app # supervisorctl -i
projections-main                 RUNNING   pid 1894, uptime 0:00:00

Am I doing something wrong or is this a bug?


supervisord -v: 3.3.5

Configuration:

[unix_http_server]
file = /run/supervisord.sock

[supervisorctl]
serverurl = unix:///run/supervisord.sock

[supervisord]
logfile = /dev/stdout
logfile_maxbytes = 0
loglevel = info
nodaemon = true
user = root

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[program:projections-main]
command = bin/console app:subscription:connect -vvv
autorestart = true
startsecs = 0
startretries = 3
stdout_logfile = /dev/stdout
stdout_logfile_maxbytes = 0
stderr_logfile = /dev/stderr
stderr_logfile_maxbytes = 0
podarcis commented 1 year ago

Any updates on this? I'm having the same issue that the program is restarted immediately (without backoff/delay) and indefinitly, without going into FATAL state.

Using startsecs=0 in combination with startretries=xx is the template configuration suggested by Symfony's Messenger Component.


supervisord -v: 3.3.1 and 4.1.0

Herz3h commented 4 months ago

same issue here, is there a workaround to this ?

restarts indefinitely

Herz3h commented 4 months ago

After debugging the code, it seems that this it the intended behavior. The startretries only applies when the process goes into BACKOFF state, not EXITED state. Now the difference is BACKOFF is when process is still starting and didn't reach state RUNNING (which is determined by startsecs).

Therefore, in my case I had set startsecs too low for symfony workers (was value 0 or 5sec), which is not enough for the process to start. And it crashed after 0 or 5 secs, which means it already reached state RUNNING. Following state is EXITED if the process crashed after reaching RUNNING state. And EXITED state doesn't apply startretries at all.

So I had to increase startsecs to 10s and now it goes into BACKOFF state and applies startretries properly.

I don't know if startretries not being applied in EXITED state is intended or not.