i-dot-ai / redbox

Bringing Generative AI to the way the Civil Service works
https://i-dot-ai.github.io/redbox/
MIT License
83 stars 27 forks source link

DRAFT: Add django-q2 as a task service #766

Closed rachaelcodes closed 1 month ago

rachaelcodes commented 1 month ago

Context

We need to schedule, and run some repeated Django tasks. Ideally we would do this in a way that allows us to view the success or otherwise of those tasks quickly, and with minimal extra overhead. I've chosen django-q2 as a fairly well-maintained and popular package that integrates well with Django, including allowing us to view and edit tasks in Django Admin.

Screenshot 2024-07-11 at 11 16 00

Changes proposed in this pull request

Adds django-q2 to the Django app with the Postgres database as the message broker. Its qcluster task monitoring service has been set up as a new Docker app. I've moved the 'delete expired data' task to a new tasks directory.

This is a draft PR Still to add:

Guidance to review

Try running this locally. To run qcluster, either run the django-q Docker app or run it from the shell in /redbox/django_app with poetry run python manage.py qcluster.

Visit Django Admin and take a look at the new Django Q2 pages. Try setting up a new scheduled task from the Django admin form. You can run the 'delete expired data' task by setting func to redbox_app.redbox_core.tasks.delete_expired_data.task.

@252afh do you think the infrastructure for this would be fairly straightforward to set up?

Relevant links

https://technologyprogramme.atlassian.net/browse/REDBOX-460

Things to check

rachaelcodes commented 1 month ago

Closing in favour of going serverless: #776 🌍