search

snakemake / snakemake-executor-plugin-slurm

A Snakemake executor plugin for submitting jobs to a SLURM cluster
MIT License
9 stars 13 forks source link

Conflicting use of arguments for global and workflow specific profiles #23

Closed TBradley27 closed 4 months ago

TBradley27 commented 5 months ago

Hello,

If I have understood the latest version of the snakemake documentation correctly ( https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles), it is best practice to have a global profile for user-specific configurations and a separate workflow-specific profiles, which is useful, for example, for resource specifications for different rules in that profile

However, both profile types make use of many of the same options. For example, it is useful to specify default_resources in the global and workflow specific profiles to respectively, specify a particular SLURM user account and partition, but also to specify a default run-time for a workflow in the workflow profile.

The problem arises when the two profiles are used together using both --profile and --workflow-profile as one profile will overwrite the other for a specific flag, in this cases default_resources. My workaround at the moment is to specify all default_resources in or the other of the global and workflow profiles. But this is not ideal, as for example, I am specifying my slurm_account in workflow profiles, whereas this would be better placed in a global profile

cmeesters commented 5 months ago

Please apologize for my delayed response. I will definitively work on the docs - as time allows.

Meanwhile, consider:

default-resources:
    slurm_account: "<my_account"
    slurm_partition: "smp"

set-resources:
    map_reads:
        cpus_per_task: 40
        mem_mb_per_cpu: 1800
        slurm_partition: "parallel"
        runtime: 60

Regardless of setting the resources on the command line or in a profile yaml, the resources override the default resources. In this example, the partition. So, I tend to have a set of default, avoiding redundancy in the rule-specific settings and proceed adding specific resources to the specific rules - notwithstanding local rules.

However, I agree the use of --profile vs. --workflow-profile is a bit confusing. Will discuss this.

TBradley27 commented 4 months ago

Hello,

Many thanks for this, and I agree that the use of set-resources can be handy here. However, I worry to the degree that this solution scales for a large number of rules. I find that setting a default number of cores for small jobs (e.g. parsing text files) is very useful, for example

cmeesters commented 4 months ago

Please check out the current docs. A minor update happened. More is to come. Suggestions are welcome.