Question: are nested supervisord instances supported?

[spiderkeys]

This may sound like a strange use-case, but I'm curious to know whether or not there is any reason why one couldn't or shouldn't run a second instance of supervisord as a child process of an upper level supervisord instance.

I've tested it out in a very basic manner, and haven't seen any issues, but was wondering if there are any core assumptions built into supervisord, when it comes to running multiple instances on a single machine, that could be being violated here.

My example conf files:

Top-level supervisord.conf

[unix_http_server]
file=/tmp/supervisor1.sock           ; the path to the socket file
chmod=0744                          ; socket file mode (default 0700)

[inet_http_server]                  ; inet (TCP) server disabled by default
port=localhost:9001                 ; ip_address:port specifier, *:port for all iface

[supervisord]
logfile=/data/logs/supervisord1.log  ; main log file; default $CWD/supervisord.log
logfile_maxbytes=10MB               ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10                  ; # of main logfile backups; 0 means none, default 10
loglevel=info                       ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord1.pid        ; supervisord pidfile; default supervisord.pid
nodaemon=true                       ; start in foreground if true; default false
minfds=1024                         ; min. avail startup file descriptors; default 1024
minprocs=200                        ; min. avail process descriptors;default 200
childlogdir=/data/logs              ; 'AUTO' child log dir, default $TEMP
identifier=supervisor1
strip_ansi=true                     ; strip ansi escape codes in logs; def. false

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor1.sock

[program:example_app]
command=bash -c "while true; do echo hello; sleep 1; done"

[program:supervisor]
command=supervisord -c supervisord2.conf

Child supervisor

[unix_http_server]
file=/tmp/supervisor2.sock           ; the path to the socket file
chmod=0744                          ; socket file mode (default 0700)

[inet_http_server]                  ; inet (TCP) server disabled by default
port=localhost:9002                 ; ip_address:port specifier, *:port for all iface

[supervisord]
logfile=/data/logs/supervisord2.log  ; main log file; default $CWD/supervisord.log
logfile_maxbytes=10MB               ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10                  ; # of main logfile backups; 0 means none, default 10
loglevel=info                       ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord2.pid        ; supervisord pidfile; default supervisord.pid
nodaemon=true                       ; start in foreground if true; default false
minfds=1024                         ; min. avail startup file descriptors; default 1024
minprocs=200                        ; min. avail process descriptors;default 200
childlogdir=/data/logs              ; 'AUTO' child log dir, default $TEMP
identifier=supervisor2
strip_ansi=true                     ; strip ansi escape codes in logs; def. false

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor2.sock

[program:example_app]
command=bash -c "while true; do echo world; sleep 1; done"

Using supervisorctl I can see the proper output from each example app under each supervisor:

supervisorctl --serverurl http://localhost:9001 tail -f example_app
==> Press Ctrl-C to exit <==
hello
^C

supervisorctl --serverurl http://localhost:9002 tail -f example_app
==> Press Ctrl-C to exit <==
world

For some specific context around what I'm trying to achieve here:

The child supervisord.conf is one that is expected to change often (manually by a user, so prone to error), and I am trying to come up with a quick method of isolating the parent instance and its direct child processes from misconfigurations in this child supervisord instance. By using an immutable configuration running in the parent instance, I expect that the parent instance will always be able to run (in the sense that it always has a valid supervisord.conf file which never changes). If someone introduces a typo in the child configuration which causes the child instance to exit altogether after a reread/reload, the parent instance should continue to run (only the child supervisord program will exit). The user could then fix whatever configuration error was made in the child conf and restart the child instance.

[mnaberez]

This may sound like a strange use-case, but I'm curious to know whether or not there is any reason why one couldn't or shouldn't run a second instance of supervisord as a child process of an upper level supervisord instance.

I've tested it out in a very basic manner, and haven't seen any issues, but was wondering if there are any core assumptions built into supervisord, when it comes to running multiple instances on a single machine, that could be being violated here.

The two supervisord instances may not share any files or sockets, which you have done. The child supervisord may not daemonize, which you have also done.

In [program:supervisord], set stopwaitsecs to a value higher than the highest stopwaitsecs value in all sections of the child supervisord's configuration. If the child supervisord does not exit before startwaitsecs, the parent supervisord will SIGKILL it, in which case the child's subprocesses will be orphaned.

I have run multiple supervisord instances on a single machine and it works. Although I've never tried running supervisord as a child of another supervisord, I can't think of any reasons why it would not work.

[spiderkeys]

Thanks for the feedback - great point on the stopwaitsecs config!