Only becomes more important when expected task execution times become more likely to overlap.
[x] Make cron task reservation scheduler
[x] Tasks to have a priority field
[x] Cron executor checks queue every 30secs and runs the next scheduled task in order of priority and time in queue, if there are no running tasks or the currently running task has flag indicating no contention for DB resources
[x] Only allow a single instance for each task in the queue, to prevent a potential backlog
We can probably re-purpose the service log for this task queue, and update the existing queries for it. We would also be able to run the longer running queries more often and make a worker pool that doesn't duplicate work.
SELECT * FROM tasksCron WHERE taskState IN (1,2,3) ORDER BY taskPriority DESC, dateQueued ASC LIMIT 0,5
Execute every 15 seconds
Grab the top 5 tasks that are either: idle, assigned, or running; ordered by priority and when they entered the queue
Update the idle tasks grabbed, setting their state to assigned
Sequentially run each of the top 5 tasks that are not blocking and are currently set to assigned
As each task finishes, set it to completed
Sequentially run each of the top 5 tasks that are blocking and are currently set to assigned, but only if there is no other blocking task running globally. If the running blocking task has been running for more than 30mins, ping the IRC
jimmydorry
commented
8 years ago
Only becomes more important when expected task execution times become more likely to overlap.
We can probably re-purpose the service log for this task queue, and update the existing queries for it. We would also be able to run the longer running queries more often and make a worker pool that doesn't duplicate work.
taskID, taskType, taskPriority, taskState, taskBlocking, dateQueued
taskTypeID, taskName (idle, assigned, running, completed)
stateID, stateName
SELECT * FROM tasksCron WHERE taskState IN (1,2,3) ORDER BY taskPriority DESC, dateQueued ASC LIMIT 0,5