search

WatsonLab / KOunt

Snakemake pipeline calculating KEGG orthologue abundance in metagenomic sequence data.
GNU General Public License v3.0
24 stars 0 forks source link

HPC configuration #2

Closed trickovicmatija closed 1 year ago

trickovicmatija commented 1 year ago

Hello,

Thank you, very nice tool! Is there any way to specify the number of threads and resources that a cluster job scheduler would also see? I am using Slurm and Cookiecutter to schedule jobs automatically. The problem is that it doesn't recognize the threads parameter that is used as a flag for a tool, but it requires a separate snakemake directive. Is there any reason why you didn't specify it there?

Best and thanks, Matija

jmattock5 commented 1 year ago

Hi Matija,

Thanks for your feedback! On our system (SGE) we create a cluster.json file in the same directory as the Snakefile specifying the default resources and resources needed for specific rules, for example:

{ "__default__" : { "core" : "1", "time" : "01:00:00", "vmem" : "1G" }, "megahit" : { "core" : "8", "time" : "48:00:00", "vmem" : "6G" }, "trim" : { "core" : "8", "time" : "08:00:00", "vmem" : "1G" }, "bwa" : { "core" : "4", "time" : "24:00:00", "vmem" : "8G" } }

We then use the following Snakemake command to refer to this file when submitting jobs:

snakemake -k --ri --use-conda --cluster-config cluster.json --cluster "qsub -R yes -V -S /bin/bash -cwd -pe sharedmem {cluster.core} -l h_rt={cluster.time} -l h_vmem={cluster.vmem}"

Hope this helps! Thanks, Jenny

jmattock5 commented 1 year ago

I'm going to close this issue, please make another if you have any more questions.