For batch processing we have been using custom task management based on cron. It had been identified to have issues (ORCIDHUB-242). Besides that there is growing need for asynchronous request handling (currently webhook event propagation to the affiliated organisations). The HUB API also currently relies on the custom task handling and that turns this piece of the HUB into one the central and critical bit.
Solution for this problems would be to add a background task queue management. The defacto standard Python task queue is Celery. The other task queue projects that arise tend to come from the perspective that Celery is overly complicated for simple use cases. Alternatives would be: The RQ (Redis Queue). Taskmaster is a lightweight simple distributed queue for handling large volumes of one-off tasks (it might be our case); Huey; Kuyruk; Dramatiq.
Making choice for the task management solution we need to take in account:
we need to provide asynchronous request handling;
scheduled tasks (task, invitation etc. expiration handling), eg, once a day;
queued multi staged tasks (invitation processing, affiliation, task, review etc. records) processing.
rpaw053
commented
6 years ago
For batch processing we have been using custom task management based on cron. It had been identified to have issues (
ORCIDHUB-242). Besides that there is growing need for asynchronous request handling (currently webhook event propagation to the affiliated organisations). The HUB API also currently relies on the custom task handling and that turns this piece of the HUB into one the central and critical bit.Solution for this problems would be to add a background task queue management. The defacto standard Python task queue is Celery. The other task queue projects that arise tend to come from the perspective that Celery is overly complicated for simple use cases. Alternatives would be: The RQ (Redis Queue). Taskmaster is a lightweight simple distributed queue for handling large volumes of one-off tasks (it might be our case); Huey; Kuyruk; Dramatiq.
Making choice for the task management solution we need to take in account:
[ORCIDHUB-290] created by rcir178