search

celery / django-celery-beat

Celery Periodic Tasks backed by the Django ORM
Other
1.68k stars 428 forks source link

Preventing duplicity while scheduling tasks with celery beat #331

Open alexandernst opened 4 years ago

alexandernst commented 4 years ago

Summary:

I have a task that I run periodically (each minute) via Celery Beat. On occasions, the task will take longer than a minute to finish it's execution, which results in the scheduler adding that task to the queue while the task is already running.

Is there a way I can avoid the scheduler adding tasks to the queue if those tasks are already running?

Exact steps to reproduce the issue:

  1. Configure a scheduled task to be executed once each minute
  2. Make the task sleep for 2 minutes.
  3. Check how the queue is growing

Detailed information

N/A

chpmrc commented 3 years ago

There is no builtin way of doing this within beat. I've been using https://github.com/cameronmaske/celery-once and it works pretty well, although it uses Redis expiring keys so it doesn't actually check whether there is already a duplicate task running.

auvipy commented 3 years ago

we might need to consider this to be in the celery core