search

a-h-b / dadasnake

Amplicon sequencing workflow heavily using DADA2 and implemented in snakemake
GNU General Public License v3.0
45 stars 17 forks source link

Slurm Latency Issues - how to add --latency-wait into config #13

Closed Fred-White94 closed 3 years ago

Fred-White94 commented 3 years ago

Hi, final issue/question (hopefully) before the pipeline is running nicely on our server.

I am running on a slurm cluster and sometimes get problems with the suggestion of adding in --latency-wait time. I am unsure where or how to add this in (presumably the slurm_simple.config.yaml file however adding the line:

  latency_wait: "--latency-wait=320"

seems to have no effect. if you could suggest where this slurm parameter could be added in that would be great.

I would also like to have some tips on the VARIABLE_CONFIG file - in particular the SUBMIT_COMMAND is unclear to me since the settings for this are declared in the config files already. When I put sbatch as the argument here it seems to try to submit the job twice.

Thanks in advance for having a look!

a-h-b commented 3 years ago

Hi, mh, concerning the latency_wait, there isn't a flexible variable at the moment which could be defined. As a work-around, you could place it in the slurm_simple.config.yaml as part of the call, as in call: "sbatch --latency-wait=320" Regarding the second question, Which of the auxiliary_files/dadasnake versions are you using and how are you calling it? You need the SUBMIT_COMMAND, if you use the auxiliary_files/dadasnake_allSubmit as dadasnake and run it with the -c option, because this will submit the top-snakemake thread in a job which has to be called using this argument. If you run dadasnake in the tmux version, it is ignored. If it submits the job twice, there might be a problem I need to check for. I hope this helps - AHB

Fred-White94 commented 3 years ago

Hi, Thanks I've tried adding the latency wait argument in the slurm config but it is in fact a snakemake argument rather than slurm.

The system has tmux installed and since it runs with the SUBMIT_COMMAND blank I'm assuming it is running on tmux. The command I am running is:

./dadasnake -c -n "MetRun"  /dadasnake/config/config.ITS1.yaml

Perhaps I could include the latency wait in the VARIABLE CONFIG file?

For now the pipeline runs without the latency-wait but I occasionally have to restart after it crashes due to missing output. Thank you. Fred.