Running list of features that should be added for SLURM

[sjeknic]

• Job history should be saved to file so that runs can be restarted part way through

• Clean up of temporary directories after (and maybe before) the runs

• Sorting of slurm logs into dated directories - with option to keep or delete (perhaps based on job status)

• Pipelines should not have to be stored as yaml files again - see if the original yaml can be used

• Input yamls should be copied to the output directory

• Killed jobs should be able to be re-started on specific partitions

• Similar to the above, but a way to prioritize certain partitions

• Generalization of the above two - but a way to arbitrarily change job submission parameters during the run

  • BUG: SLURMController doesn't stop after all Pipelines are submitted

[sjeknic]

Related to that last bug listed: It works fine if all Pipelines complete. It keeps checking if some are Failed. That's probably fine honestly.