search

rhysnewell / aviary

A hybrid assembly and MAG recovery pipeline (and more!)
GNU General Public License v3.0
82 stars 12 forks source link

coverm failure #3

Closed wwood closed 3 years ago

wwood commented 3 years ago 
(/home/woodcrob/e/binsnek-dev) cl5n012:20210108:~/m/wierdbin1/ben/57_sample_binning/2_binsnek$ binsnek recover --assembly ~/m/wierdbin1/coassembly/coassembly.spades/scaffolds.fasta --paired_reads_1 ~/m/wierdbin1/coassembly/[78]*R1.fastq.gz --paired_reads_2 ~/m/wierdbin1/coassembly/[78]*R2.fastq.gz --output . 
01/08/2021 01:27:46 PM INFO: Time - 13:27:46 08-01-2021
01/08/2021 01:27:46 PM INFO: Command - /home/woodcrob/e/binsnek-dev/bin/binsnek recover --assembly /home/woodcrob/m/wierdbin1/coassembly/coassembly.spades/scaffolds.fasta --paired_reads_1 /home/woodcrob/m/wierdbin1/coassembly/788.normal.R1.fastq.gz /home/woodcrob/m/wierdbin1/coassembly/794.normal.R1.fastq.gz /home/woodcrob/m/wierdbin1/coassembly/795.normal.R1.fastq.gz /home/woodcrob/m/wierdbin1/coassembly/805.normal.R1.fastq.gz --paired_reads_2 /home/woodcrob/m/wierdbin1/coassembly/788.normal.R2.fastq.gz /home/woodcrob/m/wierdbin1/coassembly/794.normal.R2.fastq.gz /home/woodcrob/m/wierdbin1/coassembly/795.normal.R2.fastq.gz /home/woodcrob/m/wierdbin1/coassembly/805.normal.R2.fastq.gz --output .
01/08/2021 01:27:46 PM INFO: Configuration file written to ./template_config.yaml
You may want to edit it using any text editor.
01/08/2021 01:27:46 PM INFO: Executing: snakemake --snakefile /home/woodcrob/git/BinSnek/binsnek/Snakefile --directory . --jobs 16 --rerun-incomplete --configfile './template_config.yaml' --nolock   --use-conda --conda-prefix ~/.conda/envs/   recover_mags   
Building DAG of jobs...
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/coverm.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/coverm.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/58a5c6e7)
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/concoct.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/concoct.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/935c2aae)
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/das_tool.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/das_tool.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/4759ad87)
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/checkm.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/checkm.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/2eef4795)
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/maxbin2.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/maxbin2.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/a9a788ef)
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/metabat2.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/metabat2.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/ce6ed2a2)
Creating conda environment /home/woodcrob/git/BinSnek/binsnek/envs/gtdbtk.yaml...
Downloading and installing remote packages.
Environment for ../../../../../../home/woodcrob/git/BinSnek/binsnek/envs/gtdbtk.yaml created (location: ../../../../../../home/woodcrob/.conda/envs/3a1e3607)
Using shell: /bin/bash
Provided cores: 16
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   checkm
    1   concoct_binning
    1   das_tool
    1   get_abundances
    1   get_bam_indices
    1   gtdbtk
    1   maxbin_binning
    1   metabat_binning_2
    1   prepare_binning_files
    1   recover_mags
    10
busco_folder does not point to a folder

[Fri Jan  8 13:53:11 2021]
rule prepare_binning_files:
    input: /home/woodcrob/m/wierdbin1/coassembly/coassembly.spades/scaffolds.fasta
    output: data/maxbin.cov.list, data/coverm.cov
    jobid: 8
    threads: 16

Activating conda environment: /home/woodcrob/.conda/envs/58a5c6e7
[2021-01-08T03:53:16Z INFO  coverm] CoverM version 0.6.0
[2021-01-08T03:53:16Z INFO  coverm] Setting single read percent identity threshold at 0.97 for MetaBAT adjusted coverage, and not filtering out supplementary, secondary and improper pair alignments
[2021-01-08T03:53:16Z INFO  coverm] Using min-covered-fraction 0%
[2021-01-08T03:53:16Z INFO  bird_tool_utils::external_command_checker] Found minimap2 version 2.17-r941 
[2021-01-08T03:53:17Z INFO  bird_tool_utils::external_command_checker] Found samtools version 1.9 
[2021-01-08T03:53:17Z INFO  coverm] Creating cache directory data/binning_bams/
[2021-01-08T03:53:17Z INFO  coverm::mapping_index_maintenance] Generating MINIMAP2_SR index for /home/woodcrob/m/wierdbin1/coassembly/coassembly.spades/scaffolds.fasta ..
[2021-01-08T03:53:45Z ERROR coverm::mapping_index_maintenance] Error when running MINIMAP2_SR index process.
[2021-01-08T03:53:45Z ERROR coverm::mapping_index_maintenance] The STDERR was: ""
[2021-01-08T03:53:45Z ERROR coverm::mapping_index_maintenance] Cannot continue after MINIMAP2_SR index failed.
Traceback (most recent call last):
  File "/work2/microbiome/wierdbin1/ben/57_sample_binning/2_binsnek/.snakemake/scripts/tmp3jxc4yev.get_coverage.py", line 103, in <module>
    for i in range(len(cov_list[0])):
IndexError: list index out of range
[Fri Jan  8 13:53:45 2021]
Error in rule prepare_binning_files:
    jobid: 8
    output: data/maxbin.cov.list, data/coverm.cov
    conda-env: /home/woodcrob/.conda/envs/58a5c6e7

RuleException:
CalledProcessError in line 94 of /home/woodcrob/git/BinSnek/binsnek/Snakefile:
Command 'source /home/woodcrob/e/woodcrob/bin/activate '/home/woodcrob/.conda/envs/58a5c6e7'; set -euo pipefail;  python /work2/microbiome/wierdbin1/ben/57_sample_binning/2_binsnek/.snakemake/scripts/tmp3jxc4yev.get_coverage.py' returned non-zero exit status 1.
  File "/home/woodcrob/git/BinSnek/binsnek/Snakefile", line 94, in __rule_prepare_binning_files
  File "/home/woodcrob/e/binsnek-dev/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Removing output files of failed job prepare_binning_files since they might be corrupted:
data/maxbin.cov.list, data/coverm.cov
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
An error occurred
Complete log: /work2/microbiome/wierdbin1/ben/57_sample_binning/2_binsnek/.snakemake/log/2021-01-08T132746.939540.snakemake.log
rhysnewell commented 3 years ago

Could you please attach the template_config that was involved in this crash?

wwood commented 3 years ago 
%YAML 1.1
---
fasta: /home/woodcrob/m/wierdbin1/coassembly/coassembly.spades/scaffolds.fasta
long_reads: none
short_reads_1:
- /home/woodcrob/m/wierdbin1/coassembly/788.normal.R1.fastq.gz
- /home/woodcrob/m/wierdbin1/coassembly/794.normal.R1.fastq.gz
- /home/woodcrob/m/wierdbin1/coassembly/795.normal.R1.fastq.gz
- /home/woodcrob/m/wierdbin1/coassembly/805.normal.R1.fastq.gz
short_reads_2:
- /home/woodcrob/m/wierdbin1/coassembly/788.normal.R2.fastq.gz
- /home/woodcrob/m/wierdbin1/coassembly/794.normal.R2.fastq.gz
- /home/woodcrob/m/wierdbin1/coassembly/795.normal.R2.fastq.gz
- /home/woodcrob/m/wierdbin1/coassembly/805.normal.R2.fastq.gz
batch_file: none
max_threads: 16
virsorter_data: /work/microbiome/db/virsorter_data_v2/virsorter-data
gtdbtk_folder: /work/microbiome/db/gtdbtk/release95/
busco_folder: /work/microbiome/db/busco
rhysnewell commented 3 years ago

Could you please run a clean or different local install of coverm on one of those samples by itself and see if it throws the same error?