search

cenkalti / kuyruk

⚙️ Simple task queue for Python
https://kuyruk.readthedocs.org/
MIT License
231 stars 17 forks source link

Kuyruk worker won't stop #64

Closed nathan30 closed 4 years ago

nathan30 commented 4 years ago

Hi,

I made a Python project (https://gitlab.com/edissyum/opencapture/opencaptureformaarch) and I use Kuyruk to enqeue process.

To add it run on startup I create a system service; here is the service file on Debian :

`[Unit] Description=Daemon for OpenCapture for Maarch

[Service] Type=simple

User=edissyum Group=edissyum UMask=0022

ExecStart=/opt/maarch/OpenCapture/scripts/service.sh

Restart=on-failure

TimeoutStopSec=300

[Install] WantedBy=multi-user.target

`

I had an issue, where the process aren't processed by Kuyruk. I didn't have any logs about an error and also, I can't stop the service. On the syslog I just had :

Nov 26 14:52:01 sauvegarde service.sh[2062]: W kuyruk.worker.shutdown:363 - Shutdown requested

Do you know where I can find any idea why Kuyruk doesn't handle the process ?

EDIT : After a long time, the service stopped and when I restart it, all the enqueued process are handled. Why I need to restart it ?

Thanks in advance

cenkalti commented 4 years ago

I can't tell why without seeing logs. You can view them via journalctl command on your system.

nathan30 commented 4 years ago

I can't tell why without seeing logs. You can view them via journalctl command on your system.

That's the problem, journalctl doesn't show me any logs, warning or error. But no more processes are handled..

cenkalti commented 4 years ago

You can try running /opt/maarch/OpenCapture/scripts/service.sh from console and see what happens.

nathan30 commented 4 years ago

You can try running /opt/maarch/OpenCapture/scripts/service.sh from console and see what happens.

Just trying, because the issue happen again, and while I'm running the SH, it launch Kuyruk normally, but the service worker still don't work and the restart of it is still very long

cenkalti commented 4 years ago

By default, systemd sends a SIGTERM signal, waits 90 seconds and sends a SIGKILL signal.

By default, Kuyruk waits to finish the currently running task when SIGTERM is received. If you want it to exit without finishing current task, you can tell systemd to send a SIGQUIT signal. See https://kuyruk.readthedocs.io/en/latest/worker.html for details.

To do this you have to add following line to your service description:

KillSignal=SIGQUIT

See https://www.freedesktop.org/software/systemd/man/systemd.kill.html for details.

nathan30 commented 4 years ago

By default, systemd sends a SIGTERM signal, waits 90 seconds and sends a SIGKILL signal.

By default, Kuyruk waits to finish the currently running task when SIGTERM is received. If you want it to exit without finishing current task, you can tell systemd to send a SIGQUIT signal. See https://kuyruk.readthedocs.io/en/latest/worker.html for details.

To do this you have to add following line to your service description:

KillSignal=SIGQUIT

See https://www.freedesktop.org/software/systemd/man/systemd.kill.html for details.

I've check and I see a lot of defunct process. The defunct process is the .sh files that launcher my python script, which add a task into kuyruk

Do you have an idea why those process are in defunct ?

cenkalti commented 4 years ago

See https://askubuntu.com/a/427222 for info about defunct processes.

Processes are defunct because probably there is bug in their parent.

You can see the process tree with pstree -alps <defunct_pid> command.