This PR introduces the new queueing functionality. Now, the /datasets/<dataset_id>/build-cog adds a task to the v1_queue table and starts a background task.
This background task checks each minute if there are less tasks marked as IS_PROCESSING in the queue, than configured in the settings (defaults to 2 concurrent jobs). If there is capacity, the COG is processed.
With each finishing background task, the process checks if there are still positions in the v1_queue_positions view. This is filtered for non-processing tasks only, along with their position and estimated wait time.
That means, that a background task will switch to another task, if a new task with higher priority has been added to the queue, while the background process waits for a minute. If the queue is empty, the process exits.
@JesJehle I might need to walk you through the logic, before you implement this into the frontend
This PR introduces the new queueing functionality. Now, the
/datasets/<dataset_id>/build-cogadds a task to thev1_queuetable and starts a background task.This background task checks each minute if there are less tasks marked as
IS_PROCESSINGin the queue, than configured in the settings (defaults to 2 concurrent jobs). If there is capacity, the COG is processed. With each finishing background task, the process checks if there are still positions in thev1_queue_positionsview. This is filtered for non-processing tasks only, along with their position and estimated wait time. That means, that a background task will switch to another task, if a new task with higher priority has been added to the queue, while the background process waits for a minute. If the queue is empty, the process exits.@JesJehle I might need to walk you through the logic, before you implement this into the frontend