search

franciscozorrilla / metaGEM

:gem: An easy-to-use workflow for generating context specific genome-scale metabolic models and predicting metabolic interactions within microbial communities directly from metagenomic data
https://franciscozorrilla.github.io/metaGEM/
MIT License
203 stars 42 forks source link

Allow non-cluster instances to run in parallel #120

Closed Xentrics closed 1 year ago

Xentrics commented 1 year ago

Currently, the submitLocal() function in metaGEM.sh executes Snakemake with -j 1, effectively ignoring parallelization. I suggest to change this line (358 in version 1.0.2) from

echo "snakemake all -j 1 -k"|bash; break;;

to

echo "snakemake all -j $njobs -c $ncores -k"|bash; break;;

To make full use of Snakemake.

franciscozorrilla commented 1 year ago

Thanks for the useful suggestion!

Xentrics commented 1 year ago

Apologies! I just realized a problem with that approach. To my big surprise, the jobs parameter is completely ignored when the cores parameter is specified at the same time (instead, there are as many jobs as cores) ...

I just read in the documentation for jobs that 'For local execution this is an alias for –cores.' So we cannot have both options in local mode, because cores overwrites jobs and vice versa.

So it should only be echo "snakemake all -j $njobs -k"|bash; break;;

franciscozorrilla commented 1 year ago

Ahh thanks for the clarification, I am now remembering the issues with implementing local job submission 🥲 To be honest I never really use local mode since I run my analyses on the cluster, but it does seem like there is interest in the userbase for this type of execution.