Flexible scheduling

[jpbruinsslot]

Context

The scheduler now references the /random endpoint from octopoes to get ooi's for rescheduling of tasks (ooi * available boefjes = tasks). To keep track of recurring tasks within the scheduler we allow for:

• a more deterministic approach to scheduling and rescheduling of tasks, so we don't rely on randomness from objects returned by octopoes.
• more fine-grained control of the scheduling of specific tasks, potential for scheduling job independent of signals from octopoes, katalogus, and octopoes random endpoint.
• allows for scheduling other tasks such as reports
• scheduling of tasks without an associated ooi, i.e. independent run boefjes
• scheduling of normalizer tasks, e.g. run a specific normalizer task at an interval regardless of a signal from bytes.
• give better insight into what tasks are scheduled and at what time, and how many times the specific task has been executed
• leads into new ranking possibilities

Proposed solution

Changes

Change 1: Schedule model

Create a new model Schedule which contains the necessary information to create a new Task. A Schedule has a 1-to-many relationship with an instanced Task. The updated entity relationship diagram for the scheduler:

erDiagram
task {
    id uuid PK
    scheduler_id str
    schedule_id uuid FK
    hash str
    priority int
    status taskstatus
    data jsonb
    created_at timestamp
    modified_at timestamp
}

schedule {
    id uuid PK
    scheduler_id str
    hash str
    data jsonb
    enabled bool
    schedule str
    tasks list[task]
    deadline_at timestamp
    created_at timestamp
    modified_at timestamp
}

task }o--|| schedule: ""

With both models looking like follows:

https://github.com/minvws/nl-kat-coordination/blob/394202903d434cbb9f950a89b403421feb44ed63/mula/scheduler/models/schema.py#L22-L41

[!NOTE] This model is in the process to be rename to Schedule.

https://github.com/minvws/nl-kat-coordination/blob/394202903d434cbb9f950a89b403421feb44ed63/mula/scheduler/models/task.py#L47-L66

The data field contains the actual data that is needed for a task runner to execute its task, in case of a BoefjeScheduler this is a BoefjeTask:

{
  "id": "",
  "boefje": {},
  "input_ooi":  "",
  "organization": ""
}

Change 2: Removal of the PrioritizedItem table and model

Removal of the PrioritizedItem model and table. By removing this, a substantial decrease in data duplication within the scheduler has been achieved. The queue is now 'materialized' by filtering on the task table based on the status of a Task, i.e. all task with the status QUEUED. In practice this means the following, when a task is created (either by manual entry, enabling of boefjes, rescheduling, etc) this task is added to the task table, and for every task that gets created a Schedule is created as well.

Example of both Task and Schedule table, here a Schedule already scheduled a Task in the past, and QUEUED a Task in the future.

Schedule table:

id	scheduler_id	hash	data	manual	enabled	schedule	deadline_at	created_at	modified_at
6f904	boefje-org0	1234	{}	true	true	0 12 * * 1	2020-01-01	2019-01-01	2020-01-01

[!NOTE] a Schedule can be manually defined by a cron expression

Task table:

id	scheduler_id	schedule_id	hash	priority	status	data	created_at	updated_at
59cb0	boefje-org0	6f904	1234	1	COMPLETED	{}	2019-01-01	2019-01-01
122e2	boefje-org0	6f904	1234	1	QUEUED	{}	2020-01-01	2020-01-01

When a task with the highest priority needs to be removed from the queue the following code is then being executed:

https://github.com/minvws/nl-kat-coordination/blob/394202903d434cbb9f950a89b403421feb44ed63/mula/scheduler/storage/pq_store.py#L16-L34

The rescheduling of tasks can be imagined as follows: the BoefjeScheduler queries the Schedule table where the deadline has passed (meaning a task needs to be scheduled now, and pushed onto the queue). A Task will be created and inserted into the Task table. A Task Runner can then retrieve a task through the server API, run the task and update the status when it finished.

Tasklist

• [x] #2295
• [x] #2296
• [x] https://github.com/minvws/nl-kat-coordination/issues/2285
• [x] #2287
• [x] #2286
• [x] #2300
• [x] #2313
• [x] #2288
• [x] #2306
• [x] #2289
• [x] #2290
• [x] #2372
• [x] #2591
• [x] #2611
• [x] #2650
• [x] #2651
• [x] #2652
• [x] #2654
• [x] #2653
• [ ] #3120
• [x] #2989
• [x] #2947
• [x] #3156
• [x] #3159
• [x] #3160
• [x] #3166
• [x] #2631
• [ ] Flexible scheduling: Write tests for test_scheduler.py
• [ ] #3198
• [ ] #2291
• [ ] #2292
• [ ] #2293
• [ ] #2656
• [x] #2294
• [ ] #3121

Pull Request

• https://github.com/minvws/nl-kat-coordination/pull/2786

Associated issues

• https://github.com/minvws/nl-kat-coordination/issues/199
• https://github.com/minvws/nl-kat-coordination/issues/191
• https://github.com/minvws/nl-kat-coordination/issues/205
• https://github.com/minvws/nl-kat-coordination/issues/1318

[stephanie0x00]

Can I add a request to this? When you have 10 failed tasks, it would be nice to select all the failed tasks and then simply hit the reschedule button instead of having to manually select all of them for rescheduling.

[jpbruinsslot]

The design has been agreed upon, with the advice to make sure that database normalization would be possible in the future