churchlab / millstone

Genome engineering and analysis software
http://churchlab.github.io/millstone/
MIT License
47 stars 19 forks source link

Aborted jobs in celery broker remain stuck on #481

Open glebkuznetsov opened 9 years ago

glebkuznetsov commented 9 years ago

Nothing is running but I'm seeing a celery.chord_unlock() message being repeated every 1 sec in the logs.

I'm not sure what the immediate repercussions of this are besides filling up the log file endlessly.

We somehow need a way to detect and clear the celery queue when there are no real jobs. How can we detect this?

glebkuznetsov commented 9 years ago

Just noting the way to clear these stuck tasks:

  1. Stop celery, e.g.

    supervisorctl stop celery
  2. From manage.py shell or in code:

    from celery.task.control import discard_all
    discard_all()
  3. Restart celery. should be clear:

    supervisorctl start celery