Run multiple worker

[nathan30]

Hi,

I found in an old document (v3.0-beta) and I found a Master subcommand. If I understand it well, it could runs multiple worker at the same time.

Is there an option like this in the latest version of Kuyruk ?

[cenkalti]

Version 1 of Kuyruk had master subcommand but it is removed in version 2 to simplify the code. I recommend using supervisord for managing multiple processes.

[nathan30]

Version 1 of Kuyruk had master subcommand but it is removed in version 2 to simplify the code. I recommend using supervisord for managing multiple processes.

It could be used with Kuyruk or it replace it ?

[cenkalti]

Yes, it can be used with Kuyruk. See http://supervisord.org/

[nathan30]

Yes, it can be used with Kuyruk. See http://supervisord.org/

Perfect, I manage to use it and it's works. Just an issue, don't really know if it's linked to Kuyruk. I have set supervisor to have 10 process in parralel. No issue, but after some tests, my 10 Kuyruk process became 20... Any ideas?

[cenkalti]

@nathan30 How do you count the number of Kuyruk processes?

[nathan30]

@nathan30 How do you count the number of Kuyruk processes?

Using htop or ps aux | grep kuyruk

[cenkalti]

Can you post the program section in supervisord config please?

[nathan30]

Can you post the program section in supervisord config please?

Here it is :

[program:OCWorker]
command=/opt/maarch/OpenCapture/scripts/service.sh
process_name=%(program_name)s_%(process_num)02d
numprocs=10
stderr_logfile=/tmp/test_%(process_num)02d_error.log

And here is the content of service.sh :

export LD_LIBRARY_PATH=/usr/local/lib/
export MAGICK_TMPDIR=/tmp/OpenCapture/
export TESSDATA_PREFIX=/usr/share/tesseract-ocr/4.00/tessdata/

cd /opt/maarch/OpenCapture/ || exit
/usr/local/bin/kuyruk --app src.main.OCforMaarch worker

[cenkalti]

Nothing seems unusual.

By any chance, do you use multiprocessing module in your task code or forking manually?

[nathan30]

Nothing seems unusual.

By any chance, do you use multiprocessing module in your task code or forking manually?

No, just use Threading process to run mutliple little function in parralel but no manual fork or multiprocessing module use :/

[nathan30]

After some other test I found something weird. When I run ps aux | grep kuyruk I have the following return : http://i.imgur.com/uclo6It.png with 10 result. But when I launch htop : http://i.imgur.com/FOft9yE.png with 14 results.. I didn't understand :joy:

[cenkalti]

Maybe you are seeing threads. By default htop also show threads. You can hide them by pressing F2, switching to "Display options" menu and selecting "Hide userland process threads".

[nathan30]

Maybe you are seeing threads. By default htop also show threads. You can hide them by pressing F2, switching to "Display options" menu and selecting "Hide userland process threads".

Okay, this works, it was thread on htop. But I didn't really understand what is this "threads" using the same command. I just tried to start my vm, just launch supervisord without running other things. And I have those "threads" running on htop, is that normal ? (Kuyruk was just launched, not run)

[cenkalti]

Yes, that's normal. Kuyruk creates several threads internally.

[nathan30]

Yes, that's normal. Kuyruk creates several threads internally.

Okay I understand. But after process all the "queue", why Kuyruk threads stay alive ?

[cenkalti]

Those thread are not related with the processing of messages from queues.

They are used to implement features such as --max-load flag and --max-run-time flag.

You can read the source code to learn more detail: https://github.com/cenkalti/kuyruk/blob/master/kuyruk/worker.py

[nathan30]

Hi,

I reopen this issue because I see something weird. I set supervisor to run 3 Kuyruk process. I manage to launch ~10 process. the 3 first are run well with Supervisor but when the process is ended, it doesn't run the other one. It is normal?

Thanks in advance

[cenkalti]

Hi @nathan30, I don't know the details of your setup. I think you should debug this yourself. If you can send me a reproducible bug report about Kuyruk I'll be happy to help.

[nathan30]

I finally manage to get it working. But I have another issue. If I restart Supervisor, it will create another X Kuyruk process. What can I do to kill the Kuyruk processes when Supervisor is stopped of restarted ?

Thanks in advance

[cenkalti]

It depends on how you restart supervisor. I presume you are using systemctl restart supervisor command.

Supervisor acts on following signals: http://supervisord.org/running.html#signal-handlers

If you kill it without waiting for it to shutdown gracefully, then you may see duplicate processes.

Also Kuyruk responds to signals in following way: https://kuyruk.readthedocs.io/en/latest/worker.html

Usually, it is a good practice to send a SIGTERM to a process to terminate itself gracefully, wait for some time, then kill it with SIGKILL if it still alive. This approach may change depending on the nature of the application.

You should look at Supervisor and systemd options to setup your system. http://supervisord.org/configuration.html#program-x-section-values (See stopsignal and stopwaitsecs options) https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStop=

[nathan30]

Thanks for the links. I looked at it and try a lot of configuration into my supervisor conf but without success, I stil have new process and the old ones each time I restart supervisor. Do you have any working example ?

Thanks

[cenkalti]

No, I don't have an example at the moment. How do you restart supervisor?

[nathan30]

I used to restart it with systemctl but finally I found supervisorctl to restart properly :)

[nathan30]

EDIT : Finally even with supervisorctl reload or restart all, the workers stay alive ...