Changes to get matomo tracking for API working

[sfisher]

IDK we may want to do some more testing on dev/stage servers with this before merging, just to verify it's doing as we think.

A lot of the code changes are in the other repository for the middleware. This was a lot of config to get them working.

Celery is a task runner for sending updates to Matomo asynchonously and uses some database tables to track the tasks.

I'm testing this on the dev machine right now but will put it on stage, also.

You can hit urls like curl -i https://ezid-dev.cdlib.org//id/ark:/88122/kgjw0052 and it should insert them into Matomo.

I think I have stopped it from double-logging items that have HTML since the Javascript widget already tracks those. I'll do some more checking to see if there are things I've forgotten to exclude. You can look at what it tracks at https://matomo.cdlib.org/ under the dev site right now.

It doesn't look like very much but it took me a long time to figure out all the pieces since the original repo didn't give a lot of information about how to set it up or requirements to really get it working and the error messages didn't help.

[sfisher]

I changed the target merge branch to develop.

[sfisher]

I'm going to have to make some more config changes since I think we need to configure and install CeleryBeat which allows scheduled tasks. Apparently with a database backend, it has to be installed and then runs periodically to clean up old tasks. This was super poorly documented, but this seems to be the case according to a stack overflow question. https://stackoverflow.com/questions/13147581/celery-task-clean-up-with-db-backend I also haven't seen it clean up yet since I installed things last week and we don't want the database tables to grow out of control.

[sfisher]

I've now added a cleanup task since Celery isn't cleaning up the kombu_messages table as I thought the CELERY_TASK_RESULT_EXPIRES would do. I waited over a day for cleanup and it never happened.

I based my cleanup code on https://github.com/mozilla/ichnaea/issues/22 but had to change a few things (to duplicate some functions and update code to use the new API for the database since their code didn't work correctly with current versions).

The easiest way to test is by deploying to dev, but for local testing, then follow this procedure:

• Get latest settings of settings.py based on the one created from the jinja template on dev.
• You may want to change the cleanup frequency to 60 seconds instead of an hour (60 60) so the task runs sooner. See ezidapp/celery.rb around like 97 where it shows `sender.add_periodic_task(60.0 60.0 . . .`
• Run the following two commands in separate terminals you can watch. I've been running them inside screen until Ashley helps us install them as daemons in systemd. (Connect to existing screen instances with screen -x and you can switch between virtual windows with CTRL-a <0-9> (0 to 9 for windows 0 to 9). Disconnect screen with CTRL-a d but leave it running in background. Create new screen window with CTRL-a c. celery -A ezidapp worker -l INFO # this starts the celery worker queue running celery -A ezidapp beat -s /apps/ezid/logs/db # Celery Beat allows task scheduling as needed for cleanup
• You should see a task like this run in the output of celery [2024-02-16 09:53:10,429: INFO/ForkPoolWorker-1] Task ezidapp.celery.cleanup_kombu_message_table[100a8e0d-6fad-4798-b3b1-eeb99bcaa765] succeeded in 1.248253042s
• Close down your two running daemons.
• You can also verify there are no tasks remaining in kombu_messages older than 1 day since they should've been cleaned up.