setting slurm_extra across clusters

[skinnider]

Another issue that's come up in this run is that by default, jobs submitted to argo will run only on two of the fourteen GPUs currently available. To run on the other 12, we usually have to specify:

#SBATCH --partition=main,hoppertest,skinniderlab

This would be an easy fix except that on della, we usually want to run on the PLI partition, and to access that we need to do:

#SBATCH --account=skinniderlab
#SBATCH --partition=pli

Finally, sometimes we also run on the main della partition, in which case we don't want to specify the two additional arguments above.

For now I am just manually editing the slurm_extra values in Snakefile_data to access specific partitions on specific clusters. This is a little clunky but seems to work fine. Is this how the two of you were managing this too, @vineetbansal @anushka255? Or is there a better/different way to do this, e.g., by setting some kind of cluster-specific profile, or by passing in additional arguments when calling snakemake?

[anushka255]

You can add an extra argument when calling snakemake:

snakemake --jobs 20 --slurm --default-resources slurm_partition=main,hoppertest,skinniderlab

This will set the same partition for all the rules. And if the individual rules require different partition, you can also override the default per rule by adding the following argument:

--set-resources <rule>:slurm_partition=<other_partition>

[skinnider]

Great, just ran this successfully, thanks!

[skinnider]

for future reference:

snakemake \
  --configfile /Genomics/skinniderlab/food-clm/clm/database=FooDB-representation=NA-metric=NA-optimizer=NA-rarefaction=1-enum_factor=30/config.yaml \
  --slurm --jobs 10 \
  --default-resources slurm_partition=main,hoppertest,skinniderlab