Dynamic numproc change

[ivan1986]

This would be a great feature - command to dynamically set the numprocs variable and span or kill procrocs when it change

[sobwoofer]

+1

[planetchili]

+1

[michaelarnauts]

Please stop this +1'ing. You're spamming 51 people every time.

Sorry for one more spam mail...

[julien6387]

I've just done this feature in my plugin: https://supvisors.readthedocs.io/en/0.10/xml_rpc.html#supvisors.rpcinterface.RPCInterface.update_numprocs

supervisorctl update_numprocs <program_name> <value>

It works dynamically on programs and event listeners. I did not test with real FastCGI programs but it has been implemented. if numprocs is greater than the current value, (numprocs - current) programs are added to Supervisor. If the autostart program option is set to true, the corresponding processes are then spawned. if numprocs is lower than the current value, the processes in excess are eventually stopped and then removed from Supervisor.

Basic configuration and tests can be found here: https://github.com/julien6387/supvisors/tree/master/supvisors/test/test_supervisor_177

It's quite more tricky than what Mike did in supervisor_twiddler and I don't believe that I have overcomplicated things. The relevant information is not stored in the Supervisor internal options structure - unless I missed it - so I had to re-realize the ServerOptions to get it without reinventing the wheel.

The only drawback in doing this from a plugin is that an event listener cannot subscribe to the new events (PROCESS, PROCESS_ADDED, PROCESS_REMOVED) in its configuration file as these events won't be recognized by Supervisor. The listener has to subscribe to EVENT (yeah, sob). It looks like adding new event types would require a - very simple - update of Supervisor itself. Unfortunately, I did not find a way to inject them from the plugin before the program sections are evaluated in the ServerOptions.

Mike, for future versions of Supervisors, what about adding a CUSTOM_EVENT_TYPE / PLUGIN_EVENT_TYPE (empty class CustomEventType / PluginEventType inheriting from Event) to support such feature ? Not perfect but that's a simple way to reduce the events. Ideally a plugin callback to be called before the program sections are evaluated would be nice.

[Dmytro-Shvetsov]

Hello, is anyone still working on this feature? Or can you suggest the most appropriate workaround to provide numproc dynamically?

[GameCharmer]

@Dmytro-Shvetsov if you're running separate config files for each process, instead of using numprocs, just duplicate your configs with _1, _2, _3, etc at the end. Stop all processes that you don't need, give yourself some room for expansion. It's super poor, but if you're managing your processes outside of supervisor, it works.

[Dmytro-Shvetsov]

@GameCharmer yeah, it does sound like a hell of a workaround. However, I truly cannot estimate on what hardware I'll run my project so the number of processes need to be scaled dynamically. Can you maybe suggest alternatives to supervisor to do so, or is there like a plug-and-play plugin for supervisor?

[GameCharmer]

@Dmytro-Shvetsov You can give Twiddler a shot, but I've never tried it successfully: https://github.com/mnaberez/supervisor_twiddler

[Dmytro-Shvetsov]

@GameCharmer

Wow, the Twiddler works like a charm. I managed to get the processes spawned dynamically with the following code:

from xmlrpc import client
from supervisor.xmlrpc import SupervisorTransport

server = client.ServerProxy('http://localhost', SupervisorTransport('', '', 'unix:///home/dmytro/supervisor.sock'))

cmd = '...my-command.sh'

workers_group_name = 'worker'

for gpu_id in range(num_devices):
    proc_group_name = f'procs_{gpu_id}'
    server.twiddler.addProgramToGroup(
        workers_group_name, 
        proc_group_name, 
        {'command': cmd,
         'autostart':'false', 
         'autorestart': 'True', 
         'logfile': '%(ENV_HOME)s/logs/mlworker_%(process_num)s.log',
         'startsecs': '10', 
         'process_name': '%(host_node_name)s_%(process_num)s',
         'numprocs': '5',
        }
    )
    server.supervisor.startProcessGroup(workers_group_name)
    server.twiddler.log(f"Added process group: {proc_group_name}", "INFO")

Thank you!