Koed00 / django-q

A multiprocessing distributed task queue for Django
https://django-q.readthedocs.org
MIT License
1.82k stars 283 forks source link

Do scheduled tasks run just once when there are multiple clusters? #577

Open spookylukey opened 3 years ago

spookylukey commented 3 years ago

This is a documentation request, specifically regarding the scheduler:

https://django-q.readthedocs.io/en/latest/architecture.html#scheduler

If I run clusters on multiple machines (e.g. consider an auto-scaling setup like Amazon Elasticbeanstalk which may start multiple versions of everything), can I be sure that a scheduled task will nonetheless run just once across all clusters/workers?

The scheduler component doesn't appear in the architecture diagram, so it's not easy to guess how this works.

This is particularly relevant as the corresponding feature in Celery ("beats" scheduler) explicitly says you have to ensure yourself that you don't run multiple instances of the beats process - https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html#introduction.

Thanks in advance.

Koed00 commented 3 years ago

While the task is created from the schedule, the schedule row is locked for update in an atomic transaction. That should in theory prevent another instance from creating the same task at the same scheduled time.