search

Snakemake-Profiles / slurm

Cookiecutter for snakemake slurm profile
MIT License
126 stars 44 forks source link

Priority / override order of parsing arguments to SLURM (sbatch) #42

Closed johnstonmj closed 4 years ago

johnstonmj commented 4 years ago

Hi @percyfal The Readme is quite clear regarding the priority given to each potential source of parameters to feed to SLURM, however, should the "Snakefile threads and resources (time, mem)" possibly be given higher priority?

As discussed in #40, I am passing parameters via --cluster-config snakemake -nq --profile slurm --cluster-config ~/.config/snakemake/slurm/cluster_config.yaml

As it operates currently, the --cluster-config file '_default_' values are overriding the per-rule values provided in the Snakefile. This does not seem ideal, as I expect '_default_' to only ever serve as a backup if other values are not provided.

Would it be possible (and desirable) to make the Snakefile resources and threads have top priority?

The correct parameters are passed if I make an additional 'rule' entry within the --cluster-config file. However, I would ideally use the profile cluster_config file to specify defaults only. I would prefer to allow individual rules in the Snakefile to specify their own resource requirements rather than needing to edit the --cluster-config file for each new rule.

Thanks for any advice!

percyfal commented 4 years ago

Hi @johnstonmj, I'm working through my back-log - has this issue been addressed (well, not here obviously, but in other issues of late)? I recall modifying the order of precedence following the priorities you mention regarding Snakefile threads and resources.

johnstonmj commented 4 years ago

Hi @percyfal

This seems to have been updated correctly.

Submitting with:

snakemake -nq --snakefile "minimal_example.smk" --profile /home/michael.johnston1/.config/snakemake_test/slurm

correctly submits jobs with resources specified as in minimal_example.smk

If no resources are specified for the rule within minimal_example.smk, then the default values from the profile's cluster_config.yaml are used.

Additionally, using the (deprecated) --cluster-config flag:

snakemake -nq --snakefile "minimal_example.smk" --profile /home/michael.johnston1/.config/snakemake_test/slurm --cluster-config /home/michael.johnston1/.config/snakemake_test/slurm/cluster_config.yaml

submits jobs with resources specified by the explicitly requested cluster-config file.

Seems to work well! And as described in the "Parsing arguments to SLURM" section of the Readme. Thank you!