stable/7.3.0 BUG celery invocation command syntax changed after v5, need to update supervisord config templates

[ekansa]

Arches branch stable/7.3.0 has celery (see: https://github.com/archesproject/arches/blob/stable/7.3.0/arches/install/requirements.txt#L7). However, the template config file for invoking celery has a command (https://github.com/archesproject/arches/blob/stable/7.3.0/arches/install/supervisor_celery_setup/my_proj_name-celeryd.conf#L6) that no longer has a valid syntax for celery.

If one runs a command like: /usr/local/bin/celery worker -A arches_proj.celery --loglevel=INFO

Celery will respond with an error message:

You are using `-A` as an option of the worker sub-command:
celery worker -A celeryapp <...>

The support for this usage was removed in Celery 5.0. Instead you should use `-A` as a global option:

To make it easier to deploy Arches using supervisord, we should update https://github.com/archesproject/arches/blob/stable/7.3.0/arches/install/supervisor_celery_setup/my_proj_name-celeryd.conf#L6 and replace the celery invocation with something like:

/usr/local/bin/celery -A arches_proj.celery worker --loglevel=INFO

[ekansa]

PS. It looks like master branch has the appropriate update already. This should probably propagate to other branches, especially the stable releases. See:

https://github.com/archesproject/arches/blob/master/arches/install/supervisor_celery_setup/my_proj_name-celeryd.conf#L6

[ekansa]

Similarly, the celerybeat needs an update.

The current template does not work (https://github.com/archesproject/arches/blob/stable/7.3.0/arches/install/supervisor_celery_setup/my_proj_name-celerybeat.conf#L6): celery beat -A [app].celery --schedule=/tmp/celerybeat-schedule --loglevel=INFO --pidfile=/tmp/celerybeat.pid

Something like this, with the arguments in a new order, works: celery -A arches_proj.celery beat --schedule=/tmp/celerybeat-schedule --loglevel=INFO --pidfile=/tmp/celerybeat.pid

[ekansa]

PS. Even with the correct syntax to invoke celery beat, supervisor will not be able to keep celery beat running properly:

/var/log/supervisor/supervisord.log

2023-02-14 23:55:55,453 INFO spawned: 'celerybeat' with pid 90
2023-02-14 23:55:55,460 WARN exited: celerybeat (exit status 127; not expected)
2023-02-14 23:55:56,461 INFO gave up: celerybeat entered FATAL state, too many start retries too quickly

RabbitMQ log:

arches_rabbitmq       | 2023-02-14 23:57:42.492868+00:00 [info] <0.1099.0> accepting AMQP connection <0.1099.0> (172.28.0.7:41774 -> 172.28.0.2:5672)
arches_rabbitmq       | 2023-02-14 23:57:42.497632+00:00 [info] <0.1099.0> connection <0.1099.0> (172.28.0.7:41774 -> 172.28.0.2:5672): user 'arches' authenticated and granted access to vhost '/'
arches_rabbitmq       | 2023-02-14 23:57:43.535504+00:00 [warning] <0.1099.0> closing AMQP connection <0.1099.0> (172.28.0.7:41774 -> 172.28.0.2:5672, vhost: '/', user: 'arches'):
arches_rabbitmq       | 2023-02-14 23:57:43.535504+00:00 [warning] <0.1099.0> client unexpectedly closed TCP connection

For a user, this sometimes results in a 503 error in the search interface (with django DEBUG=False). I haven't experienced a server-side error when running exactly the same setup on my localhost with DEBUG=True.

Please also note that @mradamcox is also noticing that celery beat fails with Arches v6.2 (but evidently more silently, without apparent impact to a user). See his forum comment: https://community.archesproject.org/t/how-to-shutdown-restart-celery-with-supervisor/1850/4?u=ekansa