error: <class 'xmlrpc.client.ProtocolError'>, <ProtocolError for 127.0.0.1/RPC2: 500 Internal Server Erro

[SudoGetBeer]

Hello,

When deploying our application, we are using the following command to restart the daemons: supervisorctl restart all

Recently we got a error sometimes: error: <class 'xmlrpc.client.ProtocolError'>, <ProtocolError for 127.0.0.1/RPC2: 500 Internal Server Error>: file: /usr/lib/python3/dist-packages/supervisor/xmlrpc.py line: 542

And in the supervisor.log file we see this error:

2023-08-31 11:46:38,593 ERRO XML-RPC response callback error:Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/supervisor/xmlrpc.py", line 78, in more
    value = self.callback()
  File "/usr/lib/python3/dist-packages/supervisor/rpcinterface.py", line 947, in allfunc
    callback = func(name, **extra_kwargs)
  File "/usr/lib/python3/dist-packages/supervisor/rpcinterface.py", line 301, in startProcess
    process.spawn()
  File "/usr/lib/python3/dist-packages/supervisor/process.py", line 213, in spawn
    self._assertInState(ProcessStates.EXITED, ProcessStates.FATAL,
  File "/usr/lib/python3/dist-packages/supervisor/process.py", line 185, in _assertInState
    raise AssertionError('Assertion failed for %s: %s not in %s' %  (
AssertionError: Assertion failed for daemon-755623_00: UNKNOWN not in EXITED FATAL BACKOFF STOPPED

Supervisor version is 4.2.1.

Our current "bugfix" for this is deleting the daemon and restarting it again by hand. Maybe someone else can help us with this.

[mnaberez]

AssertionError: Assertion failed for daemon-755623_00: UNKNOWN not in EXITED FATAL BACKOFF STOPPED

It looks like a process running under supervisord was in the UNKNOWN state at the time this command was sent. Running supervisorctl status should have shown UNKNOWN. A process should never enter the UNKNOWN state.

As far as I can tell, there are only two places in Supervisor 4.2.1 where a process can enter the UNKNOWN state. Both occur when sending a signal to a process (either with supervisorctl stop or supervisorctl signal).

https://github.com/Supervisor/supervisor/blob/b7ed14e100904a78c2912d39da46ef956bfbb56f/supervisor/process.py#L465-L472

https://github.com/Supervisor/supervisor/blob/b7ed14e100904a78c2912d39da46ef956bfbb56f/supervisor/process.py#L504-L511

You should see the messages above in the log file.

If this is what is happening, it could be configuration error. supervisord started a process via fork/exec but now the OS won't allow supervisord to send a signal to that process. One way this could happen is if supervisord is running as non-root and there is something like command=sudo /bin/cat in the config file.

[SudoGetBeer]

Hey @mnaberez, thanks for your reply. Now I found this error in the log:

2023-08-31 11:38:23,545 CRIT unknown problem killing daemon-755623_00 (2992376):Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/supervisor/process.py", line 466, in kill
    options.kill(pid, sig)
  File "/usr/lib/python3/dist-packages/supervisor/options.py", line 1302, in kill
    os.kill(pid, signal)
ProcessLookupError: [Errno 3] No such process

To be honest... Don't really know why this is happening.

The config is:

[program:daemon-755623]
directory=/home/xxxxx/xxxxxxxxxxxxxxx/
command=php8.2 artisan horizon

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=1
startsecs=1
redirect_stderr=true
stdout_logfile=/home/xxxx/.xxxx/daemon-755623.log
stopwaitsecs=10
stopsignal=SIGTERM
stopasgroup=true
killasgroup=true

[mnaberez]

Now I found this error in the log:

2023-08-31 11:38:23,545 CRIT unknown problem killing daemon-755623_00 (2992376):Traceback (most recent call last): File "/usr/lib/python3/dist-packages/supervisor/process.py", line 466, in kill options.kill(pid, sig) File "/usr/lib/python3/dist-packages/supervisor/options.py", line 1302, in kill os.kill(pid, signal) ProcessLookupError: [Errno 3] No such process

The process exited immediately beforesupervisord tried to kill it and supervisord did not handle the error from the OS correctly. Your Supervisor version is 4.2.1. This is a bug that was fixed in 4.2.2 (changelog, diff). Please upgrade to a newer version.