search

cameronmaske / celery-once

Celery Once allows you to prevent multiple execution and queuing of celery tasks.
https://pypi.python.org/pypi/celery_once/
BSD 2-Clause "Simplified" License
661 stars 90 forks source link

celey_once does not work #80

Closed EyecatWang closed 5 years ago

EyecatWang commented 5 years ago

I set up a task with celecy. Repeated multiple times. so i used celecy-once. but also can't. wechatimg9

wechatimg10

cameronmaske commented 5 years ago

@BlingHong Could you post the full code of the task and how you are running the task? Also, could you include which version of python + celery you are using?

KyrumX commented 5 years ago

Hi @cameronmaske

We are running into kinda the same issue as @EyecatWang , the tasks are 'sent' but nothing happens.

We are using:

celery_1     | [2018-12-18 13:29:45,747: WARNING/ForkPoolWorker-1] start supplier import task
django_1     | Performing system checks...
django_1     |
django_1     | System check identified no issues (0 silenced).
django_1     | December 18, 2018 - 13:29:48
django_1     | Django version 2.0.6, using settings 'config.settings'
django_1     | Starting development server at http://0.0.0.0:8000/
django_1     | Quit the server with CONTROL-C.
django_1     | Quit the server with CONTROL-C.
scheduler_1  | [2018-12-18 13:29:55,116: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
celery_1     | [2018-12-18 13:29:55,121: WARNING/ForkPoolWorker-2] start supplier import task
scheduler_1  | [2018-12-18 13:30:05,116: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:30:15,117: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:30:25,117: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:30:35,118: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:30:45,118: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:30:55,118: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:31:05,118: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:31:15,119: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:31:25,119: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:31:35,120: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:31:45,120: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:31:55,121: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:32:05,121: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:32:15,121: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
^[scheduler_1  | [2018-12-18 13:32:25,121: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:32:35,122: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:32:45,122: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:32:50,130: INFO/MainProcess] Writing entries...
scheduler_1  | [2018-12-18 13:32:55,122: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:33:05,123: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:33:15,124: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:33:25,124: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:33:35,125: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:33:45,125: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)
scheduler_1  | [2018-12-18 13:33:55,125: INFO/MainProcess] Scheduler: Sending due task Suppliers (biko.supplier.tasks.pull_supplier_data)

It only happens to QueueOnce tasks. But when it happens all other tasks will not work because of it. When we remove all tasks which use QueueOnce from the celery beat table and restart the 'normal' tasks run fine.

Important note is that QueueOnce tasks do work sometimes. If I restart the server a couple of times the QueueOnce tasks will run, but every now and then they just don't and the server needs to be restarted.

cameronmaske commented 5 years ago

@KyrumX @EyecatWang Are the tasks you are trying run periodic tasks, i.e. scheduled by django celery beat? If so #72 is maybe related?

This one is a bit issue to debug further. Any chance you could provide a minimal example project that reproduces the issue?

cameronmaske commented 5 years ago

Closing this out due to inactivity. If this is still an issue (or someone else encounters it) feel free comment and we can re-open.