search

adamkewley / jobson

A platform for transforming command-line applications into a job service.
Apache License 2.0
256 stars 20 forks source link

Add execution triggers #47

Closed mtazzari closed 5 years ago

mtazzari commented 6 years ago

As a zero-th order implementation of dynamic job scheduling, a very useful feature would be the concept of triggers, i.e.

job1 is submitted -> gets a jobID=ID1. job2 is submitted with triggered_by=ID1

so that: job1 may start whenever possible, but job2 is queued until job1 is finished. At that point, job2 gets next in line in the execution. This would be tremendously useful in the following scenario: job1 is a very intensive job (e.g. a fit), and job2 is some quick post-processing of the fit's result. Typically you need to repeat job2 many times for the same job1 output (e.g. to improve plots, statistical analysis etc). Since job2 can only start after job1, as of jobson 0.0.22 you need to wait job1 to finish before submitting job2 otherwise job2 would fail. Therefore it happens that you submit job1, and after some hours you are able to submit job2. In the meantime several job1 have been submitted, and the post-processing job2 has a long wait to be executed.

This is a particular case of a more general implementation of jobs' priorities. Even without reaching extreme granularity (each job with its own priority), it would be useful to have classes of jobs with different priorities set at the config.yml level.

mtazzari commented 6 years ago

This is possibly related to #23 although I don't think hooks are suitable as triggers.

adamkewley commented 5 years ago

This feature will be delayed until after 1.0.0 is released.