search

ablab / spades

SPAdes Genome Assembler
http://ablab.github.io/spades/
Other
745 stars 135 forks source link

SPAdes-3.15.2 not working as well #690

Closed amanpruthi15 closed 3 years ago

amanpruthi15 commented 3 years ago

I did upgradeto SPAdes 3.15.2 but still running into an error: == Error == system call for: "['/home/apruthi/SPAdes-3.15.2-Linux/bin/spades-hammer', '/lustre/scratch/apruthi/spades/spades_filtered/corrected/configs/config.info']" finished abnormally, OS return value: 12

I have NovaSeq Illumina reads. I have ran fastq_pair as well. Also tried --only-assembly with --careful still ran into an error.

spades.log Command line: /home/apruthi/SPAdes-3.15.2-Linux/bin/spades.py --nanopore /lustre/scratch/joh97948/bryum_filtered.fq -1 /lustre/scratch/apruthi/fastp_DNA/Female_R1_paired.fq -2 /lustre/scratch/apruthi/fastp_DNA/Female_R2_paired.fq -t 128 --careful -o /lustre/scratch/apruthi/spades/spades_filtered

System information: SPAdes version: 3.15.2 Python version: 3.7.3 OS: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core

Output dir: /lustre/scratch/apruthi/spades/spades_filtered Mode: read error correction and assembling Debug mode is turned OFF

Dataset parameters: Standard mode For multi-cell/isolate data we recommend to use '--isolate' option; for single-cell MDA data use '--sc'; for metagenomic data use '--meta'; for RNA-Seq use '--rna'. Reads: Library number: 1, library type: nanopore left reads: not specified right reads: not specified interlaced reads: not specified single reads: ['/lustre/scratch/joh97948/bryum_filtered.fq'] merged reads: not specified Library number: 2, library type: paired-end orientation: fr left reads: ['/lustre/scratch/apruthi/fastp_DNA/Female_R1_paired.fq'] right reads: ['/lustre/scratch/apruthi/fastp_DNA/Female_R2_paired.fq'] interlaced reads: not specified single reads: not specified merged reads: not specified Read error correction parameters: Iterations: 1 PHRED offset will be auto-detected Corrected reads will be compressed Assembly parameters: k: automatic selection based on read length Repeat resolution is enabled Mismatch careful mode is turned ON MismatchCorrector will be used Coverage cutoff is turned OFF Other parameters: Dir for temp files: /lustre/scratch/apruthi/spades/spades_filtered/tmp Threads: 128 Memory limit (in Gb): 250

======= SPAdes pipeline started. Log can be found here: /lustre/scratch/apruthi/spades/spades_filtered/spades.log

/lustre/scratch/apruthi/fastp_DNA/Female_R1_paired.fq: max reads length: 251 /lustre/scratch/apruthi/fastp_DNA/Female_R2_paired.fq: max reads length: 251

Reads length: 251

Default k-mer sizes were set to [21, 33, 55, 77, 99, 127] because estimated read length (251) is equal to or greater than 250

===== Before start started.

===== Read error correction started.

===== Read error correction started.

== Running: /home/apruthi/SPAdes-3.15.2-Linux/bin/spades-hammer /lustre/scratch/apruthi/spades/spades_filtered/corrected/configs/config.info

0:00:00.001 1M / 16M INFO General (main.cpp : 75) Starting BayesHammer, built from refs/heads/spades_3.15.2, git revision aab988a9b4986906b38396da7233bb1ee02982f2 0:00:00.057 1M / 16M INFO General (main.cpp : 76) Loading config from /lustre/scratch/apruthi/spades/spades_filtered/corrected/configs/config.info 0:00:00.120 1M / 16M INFO General (main.cpp : 78) Maximum # of threads to use (adjusted due to OMP capabilities): 128 0:00:00.128 1M / 16M INFO General (memory_limit.cpp : 48) Memory limit set to 250 Gb 0:00:00.140 1M / 16M INFO General (main.cpp : 86) Trying to determine PHRED offset 0:00:00.159 1M / 16M INFO General (main.cpp : 92) Determined value is 33 0:00:00.166 1M / 16M INFO General (hammer_tools.cpp : 38) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ] 0:00:00.183 1M / 16M INFO General (main.cpp : 113) Size of aux. kmer data 24 bytes === ITERATION 0 begins === 0:00:00.201 1M / 16M INFO General (kmer_index_builder.hpp : 243) Splitting kmer instances into 16 files using 128 threads. This might take a while. 0:00:00.210 1M / 16M INFO General (file_limit.hpp : 32) Open file limit set to 1024 0:00:00.232 1M / 16M INFO General (kmer_splitter.hpp : 93) Memory available for splitting buffers: 0.651041 Gb 0:00:00.244 1M / 16M INFO General (kmer_splitter.hpp : 101) Using cell size of 4194304 0:00:00.630 73G / 73G INFO K-mer Splitting (kmer_data.cpp : 97) Processing /lustre/scratch/apruthi/fastp_DNA/Female_R1_paired.fq 0:02:00.134 73G / 89G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 13392744 reads 0:04:01.802 73G / 96G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 27915447 reads 0:06:24.291 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 45019704 reads 0:08:35.162 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 60599564 reads 0:10:43.414 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 75865763 reads 0:13:14.147 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 90973825 reads 0:15:31.087 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 107102868 reads 0:17:34.787 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 121464369 reads 0:19:41.854 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 136607889 reads 0:21:38.431 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 150203469 reads 0:24:03.274 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 164621232 reads 0:24:03.299 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 97) Processing /lustre/scratch/apruthi/fastp_DNA/Female_R2_paired.fq 0:26:08.969 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 177888417 reads 0:28:32.335 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 194489782 reads 0:42:43.448 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 283402562 reads 0:49:32.044 73G / 113G INFO K-mer Splitting (kmer_data.cpp : 112) Total 329242464 reads processed 0:49:32.055 1M / 113G INFO General (kmer_index_builder.hpp : 249) Starting k-mer counting. 0:49:32.244 1M / 113G ERROR General (mmapped_reader.hpp : 52) mmap(2) failed. Reason: Cannot allocate memory. Error code: 12

== Error == system call for: "['/home/apruthi/SPAdes-3.15.2-Linux/bin/spades-hammer', '/lustre/scratch/apruthi/spades/spades_filtered/corrected/configs/config.info']" finished abnormally, OS return value: 12

In case you have troubles running SPAdes, you can write to spades.support@cab.spbu.ru or report an issue on our GitHub repository github.com/ablab/spades Please provide us with params.txt and spades.log files from the output directory.

SPAdes log can be found here: /lustre/scratch/apruthi/spades/spades_filtered/spades.log

Thank you for using SPAdes!

asl commented 3 years ago

The log still reads:

0:49:32.244 1M / 113G ERROR General (mmapped_reader.hpp : 52) mmap(2) failed. Reason: Cannot allocate memory. Error code: 12

You are still running out of RAM

chichizhao commented 3 years ago

same problem, did you solve it and how ? and you mean that the RAM is not enough for this task ?

here is my problem log.

Command line: /home/chichi/anaconda3/envs/biosoft/bin/spades.py --pe1-1 /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_1P.fq --pe1-2 /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_2P.fq -o /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades -t 50 -m 400

System information: SPAdes version: 3.15.2 Python version: 3.6.11 OS: Linux-3.10.0-1160.6.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core

Output dir: /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades Mode: read error correction and assembling Debug mode is turned OFF

Dataset parameters: Standard mode For multi-cell/isolate data we recommend to use '--isolate' option; for single-cell MDA data use '--sc'; for metagenomic data use '--meta'; for RNA-Seq use '--rna'. Reads: Library number: 1, library type: paired-end orientation: fr left reads: ['/home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_1P.fq'] right reads: ['/home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_2P.fq'] interlaced reads: not specified single reads: not specified merged reads: not specified Read error correction parameters: Iterations: 1 PHRED offset will be auto-detected Corrected reads will be compressed Assembly parameters: k: automatic selection based on read length Repeat resolution is enabled Mismatch careful mode is turned OFF MismatchCorrector will be SKIPPED Coverage cutoff is turned OFF Other parameters: Dir for temp files: /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades/tmp Threads: 50 Memory limit (in Gb): 400

======= SPAdes pipeline started. Log can be found here: /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades/spades.log

/home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_1P.fq: max reads length: 150 /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_2P.fq: max reads length: 150

Reads length: 150

Default k-mer sizes were set to [21, 33, 55, 77] because estimated read length (150) is equal to or greater than 150

===== Before start started.

===== Read error correction started.

===== Read error correction started.

== Running: /home/chichi/anaconda3/envs/biosoft/bin/spades-hammer /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades/corrected/configs/config.info

0:00:00.000 1M / 10M INFO General (main.cpp : 75) Starting BayesHammer, built from N/A, git revision N/A 0:00:00.025 1M / 10M INFO General (main.cpp : 76) Loading config from /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades/corrected/configs/config.info 0:00:00.040 1M / 10M INFO General (main.cpp : 78) Maximum # of threads to use (adjusted due to OMP capabilities): 50 0:00:00.042 1M / 10M INFO General (memory_limit.cpp : 48) Memory limit set to 400 Gb 0:00:00.042 1M / 10M INFO General (main.cpp : 86) Trying to determine PHRED offset 0:00:00.043 1M / 10M INFO General (main.cpp : 92) Determined value is 33 0:00:00.044 1M / 10M INFO General (hammer_tools.cpp : 38) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ] 0:00:00.044 1M / 10M INFO General (main.cpp : 113) Size of aux. kmer data 24 bytes === ITERATION 0 begins === 0:00:00.045 1M / 10M INFO General (kmer_index_builder.hpp : 243) Splitting kmer instances into 16 files using 50 threads. This might take a while. 0:00:00.045 1M / 10M INFO General (file_limit.hpp : 32) Open file limit set to 1024 0:00:00.045 1M / 10M INFO General (kmer_splitter.hpp : 93) Memory available for splitting buffers: 2.66667 Gb 0:00:00.046 1M / 10M INFO General (kmer_splitter.hpp : 101) Using cell size of 4194304 0:00:00.166 29G / 29G INFO K-mer Splitting (kmer_data.cpp : 97) Processing /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_1P.fq 0:01:33.577 29G / 48G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 12428463 reads 0:03:33.924 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 25007568 reads 0:05:30.243 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 37172813 reads 0:07:24.183 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 48947413 reads 0:09:17.139 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 61069698 reads 0:11:06.590 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 72855130 reads 0:13:00.091 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 84916086 reads 0:14:54.008 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 96889965 reads 0:16:42.544 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 108870243 reads 0:18:37.359 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 120939163 reads 0:20:31.552 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 132858822 reads 0:22:25.549 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 144961756 reads 0:24:18.958 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 157014129 reads 0:34:17.088 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 97) Processing /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033_2P.fq 0:40:46.991 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 290233851 reads 1:06:01.598 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 107) Processed 550542935 reads 1:17:24.866 29G / 49G INFO K-mer Splitting (kmer_data.cpp : 112) Total 663281136 reads processed 1:17:24.892 1M / 49G INFO General (kmer_index_builder.hpp : 249) Starting k-mer counting. 1:17:27.827 1M / 49G ERROR General (mmapped_reader.hpp : 52) mmap(2) failed. Reason: Cannot allocate memory. Error code: 12 1:17:27.827 1M / 49G ERROR General (mmapped_reader.hpp : 52) mmap(2) failed. Reason: Cannot allocate memory. Error code: 12

== Error == system call for: "['/home/chichi/anaconda3/envs/biosoft/bin/spades-hammer', '/home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades/corrected/configs/config.info']" finished abnormally, OS return value: 12

In case you have troubles running SPAdes, you can write to spades.support@cab.spbu.ru or report an issue on our GitHub repository github.com/ablab/spades Please provide us with params.txt and spades.log files from the output directory.

SPAdes log can be found here: /home/chichi/data/PRJNA686418/after_trimmomatic/paird_gz/SRR13376033/spades/spades.log

Thank you for using SPAdes!