search

Bogdanp / django_dramatiq

A Django app that integrates with Dramatiq.
https://dramatiq.io
Other
331 stars 77 forks source link

A lot of duplicated tasks executions #107

Closed toti1212 closed 3 years ago

toti1212 commented 3 years ago

First of all thanks for this implementation!

I'm currently running on docker-compose on an 8 core instance (docker_serivce, dramatiq_service, rabbitmq_service, postgres_service, etc): django_apscheduler==0.5.2 dramatiq [rabbitmq, watch]==1.8.1 RabbitMQ (rabbitmq: 3.8-management-alpine image)

By default, I run the command runramatiq without specifying the number of threads or processes. I understand that it will take 8 by default (depending on the implementation)

The bug is that I see is that many processes are executing the same task. The task is created only once, but several processes try to execute it, and some threads are the same between different processes.

Do you know why this could be happening? Does it have something to do with the number of threads and processes?

Here is an example. The job is in the queue once but when it's executed, many processes take it. 139698903181056 is the thread and 21 the process

"INFO | 10/06/2021 10:30:09 | 139698903181056 | app.orders.tasks | tasks | tasks.py | 22 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:09 | 139698894788352 | app.orders.tasks | tasks | tasks.py | 22 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:08 | 139698911573760 | app.orders.tasks | tasks | tasks.py | 18 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:07 | 139699123865344 | app.orders.tasks | tasks | tasks.py | 19 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:03 | 139698903181056 | app.orders.tasks | tasks | tasks.py | 21 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:02 | 139698374702848 | app.orders.tasks | tasks | tasks.py | 16 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:01 | 139698903181056 | app.orders.tasks | tasks | tasks.py | 19 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:01 | 139698903181056 | app.orders.tasks | tasks | tasks.py | 21 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:01 | 139698307593984 | app.orders.tasks | tasks | tasks.py | 21 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:00 | 139698861217536 | app.orders.tasks | tasks | tasks.py | 22 | 39 | send_menu | Sending menu #87..."
"INFO | 10/06/2021 10:30:00 | 139698894788352 | app.orders.tasks | tasks | tasks.py | 21 | 39 | send_menu | Sending menu #87..."
Bogdanp commented 3 years ago

I would make sure the task is only enqueued once. Try to use the RMQ management plugin to see the contents of the queue and maybe make sure the task isn't failing and is being retried. If you can come up with a s mall example that reproduces the issue, I can take a look. By design, though, RMQ will not lease the same message to multiple workers.

toti1212 commented 3 years ago

Thanks for answer! The problem was resolved and was not related to dramatiq nor django_dramatiq.

It seemed that the instance of django-apscheduler was executed more than once with the initiated programmer (9 times, 8 in the Dramatiq container, and 1 in another service, which is correct) by a .resume() hidden, what made it start the processes at the Dramatiq container. Then, when the date arrived, the 9 workers woke up (not the only one who would want) and the workers of Dramatiq tried to take those tasks to execute them, which caused repeated ones.