Workflow error

[Sbsten]

Hi all, I've encountered a recurring issue when using the dadasnake wrapper. The job stops after a few seconds and I end up with the following error in the slurm report (.out file) after running my personal config file: ./dadasnake -c -r config/config.16S.seb.yaml

_bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell Building DAG of jobs... Creating report... Missing metadata for file workflow.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file primers.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file dada.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file taxonomy.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file postprocessing.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file sequenceTables/all.seqTab.biom. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file preprocessing/1/NS10_Pminus_1_71610-A01_GAACTGAGCGCGCTCCACGA_L001_R1_001_AHC3HTDRXY.filt.fastq.gz.fwd.fastq. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. [...] WorkflowError: File reporting/readNumbers.tsv marked for report but does not exist. File "/home/sjaupit/personal/dadasnake/conda/snakemake_env/lib/python3.7/site-packages/snakemake/report/init.py", line 629, in auto_report_

Dadasnake is set up to submit the process to a cluster. No particular error seem to show up when doing a dry run. The config file parameters and the sample_table don't seem be the cause of the error. I've tried to remove, reinstall and rerun dadasnake in different directories and still get the same error. All output files and directory were deleted after each run.

I would greatly appreciate some insights or ideas on what could be the source of the issue here. Thanks in advance !

[a-h-b]

Hi - can you confirm that you get any output from the actual pipeline, like does it create and fill directories? -A

[Sbsten]

Hi, Yes the output folder was created and contained only an empty tmp folder and full.config.yaml -S

[a-h-b]

Ah, okay, so it did not actually do what it was supposed to do. It's a bit weird that it does not throw a proper error. If you don't mind, would you post your config file? -A

[Sbsten]

config.16S.seb.txt

Here is the config file I'm using. Although, I have some new development: I tried to run it once again today and it went further in the process, I end up with filtered, logs, preprocessing, reporting and tmp folder. But I end up with a similar workflow error in the .out file :

_File reporting/finalNumbers_perSample.tsv marked for report but does not exist. File "/home/sjaupit/dadasnake/conda/snakemake_env/lib/python3.7/site-packages/snakemake/report/init.py", line 629, in autoreport

The stderr output file gives me this:

_Error executing rule combine_or_rename on cluster (jobid: 8, external: Submitted batch job 50416, jobscript: /home/sjaupit/dadasnake/output_16S_sand1/.snakemake/tmp.4kveec6q/snakejob.combine_orrename.8.sh). For error details see the cluster log and the log files of the involved rule(s).

I'm starting to believe this could be an issue with our server

[a-h-b]

okay, so maybe, if you run it without the -r flag, at least you won't get the error from the reporting (which will anyway only work, if you have a complete run), and currently it seems to break in the middle... I'll check the config file now

[Sbsten]

I've tried running without report but it still doesn't go through

[a-h-b]

so, one thing I can tell you right now (which is not causing your error, but you'll have to change it) is that your settings will not work: you have an amplicon of > 400 bp (after primer removal) but your trunc_length and overlap settings imply only 280 bp length. You'll have to set higher numbers for trunc_length and possibly adjust the max_EE to let more reads through the quality filter. And, I think, the database you mentioned for the mothur classifier is actually a dada2 database, so you'd have to enable dada classification instead of mothur and put that file name under that category in the config. None of this is the reason for your problem, though. For that, I'd need you to check, if there is anything in the files starting with "combine_or_rename." in the "logs" folder in your output.

[Sbsten]

Thanks a lot for the suggestions, I'm changing it right now. There is no such file in the logs folder, only cutadapt.***.log

[a-h-b]

oh, that's weird, because this part actually says that it should exist, because it is where the pipeline failed.

Error executing rule combine_or_rename on cluster (jobid: 8, external: Submitted batch job 50416, jobscript: /home/sjaupit/dadasnake/output_16S_sand1/.snakemake/tmp.4kveec6q/snakejob.combine_or_rename.8.sh). For error details see the cluster log and the log files of the involved rule(s).

[a-h-b]

is there a file with "combine_or_rename" and suffix "stdout" directly in the output folder?

[Sbsten]

I re-ran it again with the report and indeed there is comine_or_rename files ending in .log in the logs folder. There's also dadasnake.combine_or_rename.***.stdout in the output directory.

[a-h-b]

cool, do either of them contain interesting news?

[Sbsten]

The .logs are empty and the .stdout contain this (sorry for the long copy paste):

_Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 2 Rules claiming more threads will be scaled down. Job counts: count jobs 1 combine_or_rename 1

[Thu Jul 1 13:38:40 2021] rule combine_or_rename: input: reporting/primerNumbers_perLibrary.tsv, preprocessing/1/A_18.fwd.fastq output: preprocessing/1/NS100_Pminus_1_71610-A09_TAATGTGTCTTATGCCTTAC_L001_R1_001_AHC3HTDRXY.filt.fastq.gz.fwd.fastq log: logs/combine_or_rename.1.NS100_Pminus_1_71610-A09_TAATGTGTCTTATGCCTTAC_L001_R1_001_AHC3HTDRXY.filt.fastq.gz.fwd.log jobid: 0 wildcards: run=1, sample=NS100_Pminus_1_71610-A09_TAATGTGTCTTATGCCTTAC_L001_R1_001_AHC3HTDRXY.filt.fastq.gz, direction=fwd resources: runtime=01:00:00, mem=8G

[Thu Jul 1 13:38:41 2021] Finished job 0. 1 of 1 steps (100%) done cluster Final resource settings: maxCores: 2 dadasnake will run Only one implementation of the bayesian classifier (default: mothur). Disable mothur to run the DADA2 implementation_

[a-h-b]

that's okay...sorry for the messy logs ;-)

[a-h-b]

but, hm, there's nothing wrong with it... does this file exist: preprocessing/1/NS100_Pminus_1_71610-A09_TAATGTGTCTTATGCCTTAC_L001_R1_001_AHC3HTDRXY.filt.fastq.gz.fwd.fastq ?

[Sbsten]

Thanks a lot for your help ! The file exists in the preprocessing folder and is not empty.

[a-h-b]

okay, so it might really be an issue with your cluster. Does your version of dadasnake already have the SNAKEMAKE_EXTRA_ARGUMENTS field in the VARIABLE_CONFIG file ?

[a-h-b]

also, do you know if you have a maximum number of jobs that you are allowed to submit to the cluster?

[Sbsten]

The SNAKEMAKE_EXTRA_ARGUMENTS field is present in the VARIABALE_CONFIG file but it's empty at the moment. I'm asking the staff responsible for the server and will let you know about the max number of job.

[a-h-b]

yes, and it might help to set SNAKEMAKE_EXTRA_ARGUMENTS to --latency-wait 10 (insert that text after one tab behind SNAKEMAKE_EXTRA_ARGUMENTS) did you use the tmux or the submit all version to run on the cluster?

[Sbsten]

Thanks, I will try that right now. I'm using submit all version to submit to the cluster, but I've tried tmux as well yesterday and it didn't seem to work at all.

[a-h-b]

re tmux: that's possible (depends on whether your cluster has/allows tmux) how many samples are you running?

[Sbsten]

We don't have a job submission limit but each node has 64 threads. I've tried to add the latency but it did not improve the output. Technically tmux is available so I'll try again.

I'm running 29 samples now as a pretest but I'll need to run >100 later.

[a-h-b]

this was related to #16 . A method to bind submission to a specific node is now implemented. I am closing this issue.