Dramatiq forks many more threads and processes than specified by flags on CLI

[pirate]

Issues

./manage.py rundramatiq --processes 1 --threads 2 --no-reload

Produces the following process tree:

What OS are you using?

This is on Ubuntu 18.04 but it's reproducible on 17.04 as well.

What version of Dramatiq are you using?

1.3.0

What did you do?

Ran a dramatiq worker with:

./manage.py rundramatiq --processes 1 --threads 2 --no-reload

What did you expect would happen?

One process would be spawned with 2 threads.

What happened?

Two processes were spawned with 6 threads.

[Bogdanp]

Interesting!

What is the output of ps aux | grep dramatiq? --processes 1 should spawn two processes: one for the master process and one worker process, but it looks like in your case there are actually 3 processes unless I'm misreading the screenshot.

Re. threads, that's expected. --threads is used to specify the number of worker threads process, but, in addition to that, dramatiq will spawn 2 consumer threads per declared queue and some middleware may also spawn background threads (e.g. the Prometheus middleware spawns one thread (only in one of the worker processes, randomly) to expose the metrics via HTTP).

[pirate]

(venv) root@blitzka-prod /o/b/grater-django# ps aux | grep dramatiq
www-data  9132  0.0  0.2 180204 24092 ?        SNl  18:41   0:01 /opt/blitzka/grater-django/venv/bin/python3.6 /opt/blitzka/grater-django/venv/bin/dramatiq --path . --processes 1 --threads 2 django_dramatiq.setup django_dramatiq.tasks grater.tasks
www-data  9140  0.1  0.8 517056 70880 ?        SNl  18:41   0:05 /opt/blitzka/grater-django/venv/bin/python3.6 /opt/blitzka/grater-django/venv/bin/dramatiq --path . --processes 1 --threads 2 django_dramatiq.setup django_dramatiq.tasks grater.tasks
root     16191  0.0  0.0  14728  1024 pts/1    R+   19:29   0:00 grep --color=auto dramatiq

Cool re: threads having two extras, that's mainly what I was wondering about.

I believe the screenshot is showing 2 processes, and 6 threads. The parent processes has one child thread and one child process. The child process has 5 child threads. (threads are green in htop)

If two processes are expected then I can safely close this issues, thanks!

[Bogdanp]

Ah, okay, so I was misreading the screenshot. It does look like you have 2 processes so I'd say this is all working as expected. All the master process does is create the worker processes, coalesce logs and wait for signals so it's not doing anything intensive if you're worried about that.

[pirate]

Cool, no worries, thanks for the quick response!

One minor suggestion, you can actually assign the threads/child proc a custom name after forking so it's better labeled in system reports like ps/top/htop: http://man7.org/linux/man-pages/man3/pthread_setname_np.3.html

Postgres uses this feature, and it's very nice when debugging to be able to figure out which process to strace.