Install and configure Celery for task tracking in our code and python config

[sfisher]

Celery is a message queue service that this library requires since tracking requests get sent to Matomo asynchronously to avoid slowdowns associated with sending tracking from the server as part of the API request. It also decouples problems with the Matomo service being down or slow from the EZID API. https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html

In addition, Celery requires some kind of persistent storage or a message broker. They prefer RabbitMQ or Redis, but it can also be run with AWS message queues, SQLite or MySQL (or others supported by SQL Alchemy library).

I believe a database version of backend is sufficient for our needs and we don't need to persistently track a large number of tasks for our use case and we can probably remove items from the list after a short time since this isn't a critical use of a message queue for an essential service.

• [x] Set up special configuration for Celery in the EZID app. (Thanks, Jing for helping me figure this out since EZID is a bit non-standard and didn't exactly match their examples).
• [x] Test with SQLite locally to ensure I see that the basics work.
• [x] Update to use MySQL since we'll need to use that when we go to multiple servers in high availability.
• [x] Update Jinga2 deploy scripts to have correct values.
• [x] Currently getting error on the EZID dev machine: ImportError: cannot import name 'Concatenate' from 'typing_extensions' (/ezid/.pyenv/versions/ezid-py38/lib/python3.8/site-packages/typing_extensions.py). which is weird since supposedly installing the library typing_extensions fixes this. It shows installed but it still doesn't work. https://stackoverflow.com/questions/70568411/cannot-import-name-concatenate-from-typing-in-3-8
• [x] Test with celery -A ezidapp worker -l INFO on dev server to be sure it works.
• [x] On first run, the celery daemon needs to run as the eziddba account since it needs to create some tables. After that it can run as ezidrw as it writes to them.
• [x] Fork and modify library so it doesn't track HTML files as API pages for those shared pages.
• [x] Allow to work with HTML 4 headers.
• [x] Add tests so they run under tox which I found was the testing framework they are using.
• [x] Real user testing to be sure it's not double-counting any non-api pages we need to exclude.
• [x] ~Figure out how to run in daemonize mode and what we need to do with Ashley to get it as a service on the servers.~ this is in another ticket now.

[ashleygould]

actually the uc3 oeprarions project is tracking this issue here directly.

I've read up on running celery in systemd based on https://docs.celeryq.dev/en/stable/userguide/daemonizing.html.

• [ ] create puppet module to manage celery service in systemd
• [ ] update puppet module to manage celerybeat service in systemd
• [ ] determine what to do about configuration
• [ ] determine if we want to run celerybeat - https://stackoverflow.com/questions/13147581/celery-task-clean-up-with-db-backend