On some very busy clusters, Cluster Flow can be slow due to it's style of submitting a separate queue job for every module. Whilst this is in theory the fastest way to process jobs because multiple steps in the same pipeline run can run side by side, in practice it can mean that the pipeline sits and waits for ten hours in the queue waiting to run a 10 second module which sends the completion e-mail.
So, to get around this, I could add a config option to choose how to parallelise. Three options: per_module (default), per_run and per_pipeline.
I have recently added functionality for Cluster Flow to run locally by running bash scripts. This behaviour could be extended to run the entire pipeline in series (per_pipeline) by submitting this bash script to the queue as a single job. For per_run, CF could write multiple bash scripts in this style and submit them each into a single queue job, with one final job containing summary modules, dependent on the others.
On some very busy clusters, Cluster Flow can be slow due to it's style of submitting a separate queue job for every module. Whilst this is in theory the fastest way to process jobs because multiple steps in the same pipeline run can run side by side, in practice it can mean that the pipeline sits and waits for ten hours in the queue waiting to run a 10 second module which sends the completion e-mail.
So, to get around this, I could add a config option to choose how to parallelise. Three options:
per_module
(default),per_run
andper_pipeline
.I have recently added functionality for Cluster Flow to run locally by running bash scripts. This behaviour could be extended to run the entire pipeline in series (
per_pipeline
) by submitting this bash script to the queue as a single job. Forper_run
, CF could write multiple bash scripts in this style and submit them each into a single queue job, with one final job containing summary modules, dependent on the others.