search

bhattlab / lathe

A tool for generating bacterial genomes from metagenomes with nanopore long read sequencing
MIT License
69 stars 18 forks source link

Snakemake fails at rule circularize_final #6

Open brymerr921 opened 5 years ago

brymerr921 commented 5 years ago

When providing Illumina and Nanopore reads, the pipeline fails at rule circularize_final because there is no directory called {sample}/3.circularization/3.circular_sequences/sh, which should have several files inside. Therefore, it fails with this error message:

519 of 524 steps (99%) done

[Thu Jun 27 23:23:58 2019]
localrule circularize_final:
    input: Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa, Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai
    output: Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta
    jobid: 528
    wildcards: sample=Prevotella-stercorea-2632

                cat Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/* | bash
                ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | cut -f1 -d '_' > circs.tmp || true
                (cat Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | grep -vf circs.tmp |
                cut -f1 | xargs samtools faidx Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa; ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | xargs cat) |
                sed 's/\([ACTG]\)\n//g' | fold -w 120 | cut -f1 -d ':' > Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta || true
                #rm circs.tmp

Activating singularity image /mnt/.snakemake/singularity/f075a3ce7bc1893be5302161e5e7807a.simg
Singularity: action-suid (U=0,P=30414)> Not mounting requested bind point (already mounted in container): /mnt/

WARNING: Not mounting requested bind point (already mounted in container): /mnt/
Singularity: action-suid (U=0,P=30414)> Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt

WARNING: Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
Singularity: action-suid (U=0,P=30414)> Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt

WARNING: Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
Singularity: action-suid (U=0,P=30414)> Skipping user bind, could not create bind point /mnt/snakemake: No such file or directory

WARNING: Skipping user bind, could not create bind point /mnt/snakemake: No such file or directory
Singularity: action-suid (U=0,P=30414)> USER=root, IMAGE='f075a3ce7bc1893be5302161e5e7807a.simg', COMMAND='exec'

cat: 'Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/*': No such file or directory
[Thu Jun 27 23:23:58 2019]
Error in rule circularize_final:
    jobid: 528
    output: Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta
    shell:

                cat Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/* | bash
                ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | cut -f1 -d '_' > circs.tmp || true
                (cat Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | grep -vf circs.tmp |
                cut -f1 | xargs samtools faidx Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa; ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | xargs cat) |
                sed 's/\([ACTG]\)\n//g' | fold -w 120 | cut -f1 -d ':' > Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta || true
                #rm circs.tmp

        (exited with non-zero exit code)

Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /mnt/.snakemake/log/2019-06-27T231855.973555.snakemake.log

This is the current directory structure:

Prevotella-stercorea-2632/
├── 0.basecall
│   ├── Prevotella-stercorea-2632.fq
│   ├── data_links
│   │   └── dummy
│   └── raw_calls
│       └── dummy
├── 1.assemble
│   ├── Prevotella-stercorea-2632_nomerge.fasta -> /mnt/Prevotella-stercorea-2632/1.assemble/assemble_4m/Prevotella-stercorea-2632_4m.contigs.corrected.fasta
│   ├── Prevotella-stercorea-2632_raw_assembly.fa
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.amb
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.ann
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.bwt
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.fai
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.pac
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.sa
│   └── assemble_4m
│       ├── Prevotella-stercorea-2632_4m.contigs.corrected.fasta
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.bam
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.bam.bai
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.fai
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.misassemblies.tsv
│       ├── Prevotella-stercorea-2632_4m.contigs.layout
│       ├── Prevotella-stercorea-2632_4m.contigs.layout.readToTig
│       ├── Prevotella-stercorea-2632_4m.contigs.layout.tigInfo
│       ├── Prevotella-stercorea-2632_4m.correctedReads.fasta.gz
│       ├── Prevotella-stercorea-2632_4m.report
│       ├── Prevotella-stercorea-2632_4m.seqStore
│       ├── Prevotella-stercorea-2632_4m.seqStore.err
│       ├── Prevotella-stercorea-2632_4m.seqStore.ssi
│       ├── Prevotella-stercorea-2632_4m.trimmedReads.fasta.gz
│       ├── Prevotella-stercorea-2632_4m.unassembled.fasta
│       ├── Prevotella-stercorea-2632_4m.unitigs.bed
│       ├── Prevotella-stercorea-2632_4m.unitigs.fasta
│       ├── Prevotella-stercorea-2632_4m.unitigs.gfa
│       ├── Prevotella-stercorea-2632_4m.unitigs.layout
│       ├── Prevotella-stercorea-2632_4m.unitigs.layout.readToTig
│       ├── Prevotella-stercorea-2632_4m.unitigs.layout.tigInfo
│       ├── canu-logs
│       ├── canu-scripts
│       ├── correction
│       ├── trimming
│       └── unitigging
├── 2.polish
│   ├── Prevotella-stercorea-2632_polished.fa
│   ├── Prevotella-stercorea-2632_polished.fa.fai
│   └── pilon
│       ├── Prevotella-stercorea-2632_pilon.fa
│       ├── corrections.vcf.gz
│       ├── corrections.vcf.gz.tbi
│       ├── ranges
│       ├── short_reads.bam
│       ├── short_reads.bam.bai
│       └── sub_runs
└── 3.circularization
    └── 1.candidate_genomes

Full log file attached: 190627__circularize_final__error.log

elimoss commented 5 years ago

I think I see what's happening. Would you mind showing me the result of sort -k2,2gr Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | head please? I think you've found an edge case that I have failed to account for.

brymerr921 commented 5 years ago

This is the output:

tig00000012 85799   743756  60  61
tig00000007 65682   616905  60  61
tig00000001_5   62620   28879   60  61
tig00000010 59063   683695  60  61
tig00000047 50189   1228806 60  61
tig00000023 50158   908211  60  61
tig00000413 46361   2760317 60  61
tig00000031 45324   1069679 60  61
tig00000003_5   45045   192121  60  61
tig00000029 43222   1025723 60  61
brymerr921 commented 5 years ago

Running the following, and then restarting the snakemake pipeline allowed the pipeline to complete.

mkdir -p Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/
touch Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/dummy
saras224 commented 3 years ago

Hi @brymerr921 Did you get the solution to your problem? Cuz I am facing a similar issue where it says : Activating singularity image /lustre/rsharma/.snakemake/singularity/f075a3ce7bc1893be5302161e5e7807a.simg Waiting at most 30 seconds for missing files. MissingOutputException in line 574 of /home/rsharma/lathe/Snakefile: Job Missing files after 30 seconds: W1/3.circularization/1.candidate_genomes This might be due to filesystem latency. If that is the case, consider increasing the wait time with --latency-wait. Job id: 25 completed successfully, but some output files are missing. 25 Job failed, going on with independent jobs. Exiting because a job execution failed. Look above for error message Complete log: /lustre/rsharma/.snakemake/log/2021-08-12T150046.064606.snakemake.log {'W1': ['/home/rsharma/WWK_forward_paired.fastq', '/home/rsharma/WWK_reverse_paired.fastq']} Checking for updates or modifications to workflow WARNING: Differences to latest version detected. Please reset changes and/or pull repo.

Saraswati

saras224 commented 3 years ago

@brymerr921 Do I have to run the whole pipeline again or I can start from circularization?