search

justanhduc / task-spooler

A scheduler for GPU/CPU tasks
https://justanhduc.github.io/2021/02/03/Task-Spooler.html
GNU General Public License v2.0
273 stars 24 forks source link

Enhancement request: support for priorities #50

Open tomaszklim opened 1 year ago

tomaszklim commented 1 year ago

Task spooler would be a great replacement for "real" queue servers, but it misses a few capabilities. One of them is support for priorities.

Minimal form: normal and high priority (or normal and low).

Ideally, 3 priorities (low, normal, high), or even more.

justanhduc commented 1 year ago

hi @tomaszklim. It can be done by manually swapping the jobs for now. Is there any other benefit for doing this?

tomaszklim commented 1 year ago

To swap jobs, you need to:

Imagine that you have millions or even billions of items to process - eg. imagine that you're processing Twitter streams and each tweet is 1 job. Is would be optimal to keep such queue as busy as possible. But sometimes (and only sometimes) you have some tasks, that individually are more important and urgent - so you want to process them quicker. As since the quantity of processed tasks is so big. that it is impossible or impractical to work with them on per-task basis.