snakemake / snakemake-executor-plugin-slurm

A Snakemake executor plugin for submitting jobs to a SLURM cluster
MIT License
18 stars 19 forks source link

How to set CommonSettings from command line? #73

Closed choosehappy closed 4 months ago

choosehappy commented 7 months ago

I nicely have everything up and running with the latest versions of snakemake and snakemake-executor-plugin-slurm pulled from github

Thanks for all your hard work!

One quick question that i can't suss out from the available documentation - how do modify the CommonSettings parameters, and in particular this one:

https://github.com/snakemake/snakemake-executor-plugin-slurm/blob/58af422542aa4a4e503d9b61123d52c01b41893a/snakemake_executor_plugin_slurm/__init__.py#L43

my current command line looks like this, but no modifications seem to be able to impact that variable:

snakemake  --executor slurm -j 20  --latency-wait 1 --default-resources  "slurm_account=root" "slurm_partition=LocalQ" mem_mb=0 --config json_file=sectra_json_example.json

I manually changed the variable from 40 down to 2, and in my test situation the computation time goes from 4 minutes down to 40 seconds -- most of it is clearly spent in waiting to begin querying for jobs which are already completed

Thanks again!

cmeesters commented 6 months ago

As this is only present in snakemake_interface_executor_plugins and code making use of it, not in the snakemake core, I too wonder how this can be tuned.

Meanwhile, I too, find this value a bit high: Particularly when teaching, but also upon development (when many jobs fail) such a default hinders progress.

So, is there a way to set this without tinkering with the code? Would a default of 5 seconds (or even lower) be more sensible? If a slurmdb, as suggested in the code comment, can take half a minute to be queried, there is something wrong with the cluster, and we might reasonably be catching issues when inquiring the job status, right?

choosehappy commented 4 months ago

Amazing, thank you!!

cmeesters commented 4 months ago

wow - you are fast. I wanted to write to you, only to find this issue already closed.

I am sorry, it took so long.

choosehappy commented 4 months ago

Yea no problem -- really ... amazingly, its coming in right on time : )