Celery on production guidelines - broken links; How to setup two queues with supervisor?

[KrzysztofMadejski]

In readme.md those links are broken:

• http://docs.ckan.org/en/latest/extensions.html#enabling-an-extension-with-background-tasks - points to 404
• http://wiki.ckan.org/Writing_asynchronous_tasks - points to empty wiki

I've checked following links:

• https://github.com/ckan/ckan/blob/master/ckan/config/celery-supervisor.conf
• http://docs.ckan.org/en/ckan-2.3.5/maintaining/background-tasks.html
• or http://docs.ckan.org/en/latest/maintaining/background-tasks.html (ckan 2.7)

but none of them specifies how to setup two queues with supervisor. @davidread do you have any working example of such supervisor config file?

BTW: Warning: in 2.7 a new background job system has been introduced. As such the archiver probably won't with new CKAN. See #42

[KrzysztofMadejski]

I've done 2 processes for celery, let me know if you have better idea.

; ===============================
; ckan celeryd supervisor example
; ===============================

; symlink or copy this file to /etc/supervisr/conf.d 
; change the path/to/virtualenv below to the virtualenv ckan is in.

[program:celery-priority]
; Full Path to executable, should be path to virtural environment,
; Full path to config file too.

command=/home/ckan/.virtualenvs/ckan/bin/paster --plugin=ckanext-archiver celeryd2 run priority --config=/etc/ckan/prod.ini

; user that owns virtual environment.
user=ckan

numprocs=1
stdout_logfile=/var/log/ckan/celeryd.log
stderr_logfile=/var/log/ckan/celeryd.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

[program:celery-bulk]
; Full Path to executable, should be path to virtural environment,
; Full path to config file too.

command=/home/ckan/.virtualenvs/ckan/bin/paster --plugin=ckanext-archiver celeryd2 run bulk --config=/etc/ckan/prod.ini

; user that owns virtual environment.
user=ckan

numprocs=1
stdout_logfile=/var/log/ckan/celeryd.log
stderr_logfile=/var/log/ckan/celeryd.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

[davidread]

Yep that's what we did - run it twice with celeryd2.

[KrzysztofMadejski]

Ok, I've added a suggested template with two definitions and updated the docs. All in #44 PR ready to be merged.