search

OLC-Bioinformatics / olc_genomics_portal

Source for OLC's Genomics Portal
3 stars 1 forks source link

Better task queue handling #4

Closed lowandrew closed 5 years ago

lowandrew commented 5 years ago

Current task queue is implemented with django-background-task, which is just a simple FIFO system. Would be good to replace with something that can handle more complex cases (maybe celery?)

lowandrew commented 5 years ago

As of https://github.com/OLC-Bioinformatics/olc_genomics_portal/commit/6217b7de47e0d22a67e05ac9ed03ffa9ce4a41ba celery now appears to be working.

TODO from here: upgrade celery version to 4.x instead of current 3.x, set up multiple queues for different priority/job size of things, get tasks switched over to celery, and then get rid of django-background-task

b-arbour commented 5 years ago

Done!