Supervisord unable to start when I run docker-compose up

[valpet93]

Hi there.

When I run sudo docker-compose up, supervisord is unable to start. It seems the problem is linked to postfix (master) and resque workers (see below).

I tried several times, installing the docker on ubuntu both server (22 and 20) and client (22 and 20).

misp_web | MISP is now live. Users can now log in.Starting supervisord
misp_web | 2022-07-27 07:38:14,592 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
misp_web | 2022-07-27 07:38:14,599 INFO supervisord started with pid 1
misp_web | 2022-07-27 07:38:15,605 INFO spawned: 'apache2' with pid 95
misp_web | 2022-07-27 07:38:15,611 INFO spawned: 'misp-modules' with pid 96
misp_web | 2022-07-27 07:38:15,633 INFO spawned: 'master' with pid 97
misp_web | 2022-07-27 07:38:15,639 INFO spawned: 'redis-server' with pid 98
misp_web | 2022-07-27 07:38:15,658 INFO spawned: 'resque' with pid 99
misp_web | 2022-07-27 07:38:15,690 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
misp_web | 2022-07-27 07:38:15,693 INFO success: master entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
misp_web | 2022-07-27 07:38:15,693 INFO success: resque entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
misp_web | 2022-07-27 07:38:15,694 INFO exited: resque (exit status 1; not expected)
misp_web | 2022-07-27 07:38:16,267 INFO exited: master (exit status 0; expected)
misp_web | 2022-07-27 07:38:17,270 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
misp_web | 2022-07-27 07:38:17,270 INFO success: redis-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

However, the docker containers are up. I can log in to MISP GUI. There are no workers shown in diagnostic. Moreover, if I set supervisor to true, no settings page is reachable, as it is supposed since it is not up.

In addition, I'm able to access to docker container with sudo docker exec -i -t 50dfd5921cfe /bin/bash command. Those below are the output of several command to check supervisor status

root@50dfd5921cfe:/var/www/MISP# sudo service supervisor stop
root@50dfd5921cfe:/var/www/MISP# sudo service supervisor start
root@50dfd5921cfe:/var/www/MISP# sudo service supervisord status
supervisord: unrecognized service
root@50dfd5921cfe:/var/www/MISP# sudo supervisorctl status
unix:///var/run/supervisor.sock no such file
root@50dfd5921cfe:/var/www/MISP#

However, I tried to figure out the issue, so I uninstalled any trace of supervisor

sudo apt-get remove supervisor
sudo apt-get remove --auto-remove supervisor
sudo apt-get purge supervisor
sudo apt-get purge --auto-remove supervisor

and installed it again following the official MISP guide with some tweaks (see below for an example): I changed the 28th line of the file /etc/supervisor/supervisord.conf from

27 [include]
28 files = /etc/supervisor/conf.d/*.conf

to

27 [include]
28 files = /etc/supervisor/conf.d/misp-workers.conf.conf

Eventually, I succeed:

root@50dfd5921cfe:/var/www/MISP/app# sudo supervisorctl status
misp-workers:cache_00            RUNNING   pid 2446, uptime 0:00:07
misp-workers:cache_01            RUNNING   pid 2464, uptime 0:00:07
misp-workers:cache_02            RUNNING   pid 2466, uptime 0:00:07
misp-workers:cache_03            RUNNING   pid 2467, uptime 0:00:07
misp-workers:cache_04            RUNNING   pid 2470, uptime 0:00:07
misp-workers:default_00          RUNNING   pid 2403, uptime 0:00:07
misp-workers:default_01          RUNNING   pid 2404, uptime 0:00:07
misp-workers:default_02          RUNNING   pid 2405, uptime 0:00:07
misp-workers:default_03          RUNNING   pid 2409, uptime 0:00:07
misp-workers:default_04          RUNNING   pid 2415, uptime 0:00:07
misp-workers:email_00            RUNNING   pid 2416, uptime 0:00:07
misp-workers:email_01            RUNNING   pid 2424, uptime 0:00:07
misp-workers:email_02            RUNNING   pid 2427, uptime 0:00:07
misp-workers:email_03            RUNNING   pid 2428, uptime 0:00:07
misp-workers:email_04            RUNNING   pid 2436, uptime 0:00:07
misp-workers:prio_00             RUNNING   pid 2472, uptime 0:00:07
misp-workers:prio_01             RUNNING   pid 2479, uptime 0:00:07
misp-workers:prio_02             RUNNING   pid 2480, uptime 0:00:07
misp-workers:prio_03             RUNNING   pid 2496, uptime 0:00:07
misp-workers:prio_04             RUNNING   pid 2502, uptime 0:00:07
misp-workers:update_00           RUNNING   pid 2509, uptime 0:00:07

BUT whenever I restart the docker container, that keeps to report the same error reported at the start of the post, I have to stop and start supervisorctl service...

There is something broken in the installation script I suppose.

What can I do for you in order to fix it?

Below you can find the whole output of the docker-compose up command as encoded file.

docker-compose up output.txt

See you

[righel]

Hi @valpet93 The guide is only required if you want to change the default way MISP workers are handled, I checked and the issue why the workers were not starting was a regression introduced by https://github.com/MISP/misp-docker/pull/140. Please pull the repo and try to build the containers again, workers should start without issues.

Let me know if this helps.

[valpet93]

Hi @righel ,

First of all I would thank you for your patience.

Actually, the fix seems to fail as the output is quite similar:

misp_web | MISP is now live. Users can now log in.Starting supervisord
misp_web | 2022-07-27 13:37:57,951 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
misp_web | 2022-07-27 13:37:57,958 INFO supervisord started with pid 1
misp_web | 2022-07-27 13:37:58,965 INFO spawned: 'apache2' with pid 75
misp_web | 2022-07-27 13:37:58,967 INFO spawned: 'misp-modules' with pid 76
misp_web | 2022-07-27 13:37:58,970 INFO spawned: 'master' with pid 77
misp_web | 2022-07-27 13:37:58,974 INFO spawned: 'redis-server' with pid 78
misp_web | 2022-07-27 13:37:58,977 INFO spawned: 'resque' with pid 79
misp_web | 2022-07-27 13:37:59,136 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
misp_web | 2022-07-27 13:37:59,136 INFO success: master entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
misp_web | 2022-07-27 13:37:59,136 INFO success: resque entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
misp_web | 2022-07-27 13:37:59,536 INFO exited: master (exit status 0; expected)
misp_web | 2022-07-27 13:37:59,992 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
misp_web | 2022-07-27 13:37:59,992 INFO success: redis-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
misp_web | 2022-07-27 13:38:02,508 INFO exited: resque (exit status 0; expected)

However, the post-fix exit status of both resque and master is 0, which is way better than before.

Any further idea to figure out the problem?

[righel]

Do you have the same output in MISP diagnostics -> workers?

[valpet93]

Actually no, they are good:

What's going on in there @righel ?

What the output of the sudo docker-compose up command is supposed to be like? By my side it seems to be not finished as no shell prompt is prompted in cli:

[valpet93]

In addition, I tried to brutally shut down the machine (no Ctrl+C to the command) and restarted. The docker at the boot of the machine was already up with MISP GUI totally working along with all the workers.

However, is the supervisor supposed to be installed by the sudo docker-compose up command?

[righel]

Hello, I'm not the original author of this repo but it works like this:

1. Supervisor is installed here: https://github.com/MISP/misp-docker/blob/master/web/Dockerfile#L12
2. The supervisor programs configuration is defined here: https://github.com/MISP/misp-docker/blob/master/web/Dockerfile#L39
3. Then supervisor is manually launched in nodaemon mode here: https://github.com/MISP/misp-docker/blob/master/web/run.sh#L145 in the docker-compose.yml entrypoint

The issue you faced originally was the workers were trying to run as root and there's a check in the script that starts the workers that prevents this, see: https://github.com/MISP/MISP/blob/2.4/app/Console/worker/start.sh#L5

From what I can see it is the expected behavior for the master and resque programs to exit. The workers are spawned after start.sh finishes execution, same happens with postfix, the daemon is also running after the master program exits.