search

MIT-LCP / physionet-build

The new PhysioNet platform.
https://physionet.org/
BSD 3-Clause "New" or "Revised" License
55 stars 19 forks source link

Add Django Q2 for task management and scheduling #2207

Closed tompollard closed 3 months ago

tompollard commented 3 months ago

Why switch to Django Q2

We have a need to be able to:

  1. Move tasks to the background, e.g. when uploading files to cloud and sending emails.
  2. Run scheduled tasks, e.g. scrubbing spam accounts, logging metrics from services like CrossRef, DataCite, Google Scholar, etc.

Right now, we have:

Django Q2 is well documented and under active development. It is compatible with Django 5 (which I hope we can move to soon-ish).

What's in this pull request

This pull request:

  1. Adds the Django Q2 dependency
  2. Adds configuration to Django settings.
  3. Sets the backend to the Django ORM (rather than external tools like Redis/Celery), for simplicity.
  4. Provides instructions for running the Django Q2 server.

After the dependency is added, I would like to:

bemoody commented 3 months ago

This all looks good and django-q2 sounds like a good way to go; thanks for find it.

Django Cron tasks that don't appear to run because Django Cron is not installed.

Just to be clear: you make it sound like we're not running any periodic background tasks at the moment. We do run periodic tasks using cron. We do not use django-cron to do so.

I'm happy to merge this so you can experiment, but please don't change the existing cron jobs until I have a chance to look at them in detail.

tompollard commented 3 months ago

Thanks!

Django Cron tasks that don't appear to run because Django Cron is not installed.

Just to be clear: you make it sound like we're not running any periodic background tasks at the moment. We do run periodic tasks using cron. We do not use django-cron to do so.

Yep, I'm not saying that we don't run cron tasks. e.g. we're purging old accounts here:

https://github.com/MIT-LCP/physionet-build/blob/83c80c59ce618cbe9c9e2c239e143db7849d88fa/deploy/production/etc/cron.d/physionet#L11-L15

My point was just that we have django-cron tasks defined in https://github.com/MIT-LCP/physionet-build/blob/dev/physionet-django/physionet/cron.py that (as far as I can see) are not being run:

https://github.com/MIT-LCP/physionet-build/blob/83c80c59ce618cbe9c9e2c239e143db7849d88fa/physionet-django/physionet/cron.py#L14-L73