kagkarlsson / db-scheduler

Persistent cluster-friendly scheduler for Java
Apache License 2.0
1.25k stars 190 forks source link

Support simple form of task priority #232

Open kagkarlsson opened 3 years ago

kagkarlsson commented 3 years ago

It would be good to support a basic form for task priority that would ensure that certain tasks would get priority if there suddenly is a long queue of due executions.

A bit worried about performance implications. If we allow priority to be an unbounded field, it will be difficult to create indexes for optimal polling.

SELECT * from scheduled_executions
WHERE execution_time > :now
ORDER BY priority asc, execution_time asc

Normally this is a perfect index for polling, but it will not be if we add priority to the ORDER BY clause:

create index on scheduled_tasks(execution_time asc);

A possible compromise is to support a small fixed set of priorities and optimize polling queries...

Desislav-Petrov commented 2 years ago

Are you interested in PRs for this one?

kagkarlsson commented 2 years ago

There is one already, though I have not had the time to analyze the performance implications of it. ( #183 )

The first thing I would do here is to create some synthetic data, say 10M executions in the table with a new field priority (varied data) and run the polling query to see how it performs. If bad, try to optimize using indices. Once we know how it performs, we can decide how the feature should be designed..

Would you like to contribute with some testing? And dump some EXPLAINs?