search

faircloth-lab / phyluce

software for UCE (and general) phylogenomics
http://phyluce.readthedocs.org/
Other
80 stars 49 forks source link

<jemalloc>: Error in malloc(): out of memory #284

Open marianamazzochi opened 2 years ago

marianamazzochi commented 2 years ago

Dear Brant, I am working with a set of 67 samples in Linux. When I ran phyluce_assembly_assemblo_spades, computer worked for a whole week and I got an output for 65 samples (folders with contigs.fasta, scaffolds.fasta and spades.log, so I assume everything went out just right); but when it got to the last two samples, I started to get this error:

2022-11-10 10:13:29,888 - phyluce_assembly_assemblo_spades - INFO - =========== Starting phyluce_assembly_assemblo_spades =========== 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Version: 1.7.1 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Commit: None 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --config: /home/aline/UCEs_Mari/assembly2.conf 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --cores: 1 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --dir: None 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --log_path: None 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --memory: 8 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --no_clean: False 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --output: /home/aline/UCEs_Mari/spades 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --subfolder: 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Argument --verbosity: INFO 2022-11-10 10:13:29,889 - phyluce_assembly_assemblo_spades - INFO - Getting input filenames and creating output directories 2022-11-10 10:13:29,890 - phyluce_assembly_assemblo_spades - INFO - ---------------------- Processing anous-66 ---------------------- 2022-11-10 10:13:29,891 - phyluce_assembly_assemblo_spades - INFO - Finding fastq/fasta files 2022-11-10 10:13:29,894 - phyluce_assembly_assemblo_spades - INFO - File type is fastq 2022-11-10 10:13:29,894 - phyluce_assembly_assemblo_spades - INFO - Running SPAdes for PE data Command line: /home/aline/.conda/envs/phyluce-1.7.1/bin/spades.py --careful --sc --memory 8 --threads 1 --cov-cutoff 5--pe1-1 /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ1.fastq.gz --pe1-2 /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ2.fastq.gz -o /home/aline/UCEs_Mari/spades/anous-66_spades --pe1-s /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ-singleton.fastq.gz

System information: SPAdes version: 3.14.1 Python version: 3.6.13 OS: Linux-5.4.0-131-generic-x86_64-with-debian-buster-sid

Output dir: /home/aline/UCEs_Mari/spades/anous-66_spades Mode: read error correction and assembling Debug mode is turned OFF

Dataset parameters: Single-cell mode Reads: Library number: 1, library type: paired-end orientation: fr left reads: ['/media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ1.fastq.gz'] right reads: ['/media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ2.fastq.gz'] interlaced reads: not specified single reads: ['/media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ-singleton.fastq.gz'] merged reads: not specified Read error correction parameters: Iterations: 1 PHRED offset will be auto-detected Corrected reads will be compressed Assembly parameters: k: [21, 33, 55] Repeat resolution is enabled Mismatch careful mode is turned ON MismatchCorrector will be used Coverage cutoff is turned ON and threshold is 5.000000 Other parameters: Dir for temp files: /home/aline/UCEs_Mari/spades/anous-66_spades/tmp Threads: 1 Memory limit (in Gb): 8

======= SPAdes pipeline started. Log can be found here: /home/aline/UCEs_Mari/spades/anous-66_spades/spades.log

/media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ1.fastq.gz: max reads length: 151 /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ2.fastq.gz: max reads length: 151 /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ-singleton.fastq.gz: max reads length: 151

Reads length: 151

===== Before start started.

===== Read error correction started.

===== Read error correction started.

== Running: /home/aline/.conda/envs/phyluce-1.7.1/bin/spades-hammer /home/aline/UCEs_Mari/spades/anous-66_spades/corrected/configs/config.info

0:00:00.000 3M / 10M INFO General (main.cpp : 75) Starting BayesHammer, built from N/A, git revision N/A 0:00:00.001 3M / 10M INFO General (main.cpp : 76) Loading config from /home/aline/UCEs_Mari/spades/anous-66_spades/corrected/configs/config.info 0:00:00.003 4M / 10M INFO General (main.cpp : 78) Maximum # of threads to use (adjusted due to OMP capabilities): 1 0:00:00.003 4M / 10M INFO General (memory_limit.cpp : 49) Memory limit set to 8 Gb 0:00:00.003 4M / 10M INFO General (main.cpp : 86) Trying to determine PHRED offset 0:00:00.005 4M / 10M INFO General (main.cpp : 92) Determined value is 33 0:00:00.005 4M / 10M INFO General (hammer_tools.cpp : 38) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ] 0:00:00.005 4M / 10M INFO General (main.cpp : 113) Size of aux. kmer data 24 bytes === ITERATION 0 begins === 0:00:00.006 4M / 10M INFO K-mer Index Building (kmer_index_builder.hpp : 301) Building kmer index 0:00:00.006 4M / 10M INFO General (kmer_index_builder.hpp : 117) Splitting kmer instances into 16 files using 1 threads. This might take a while. 0:00:00.007 4M / 10M INFO General (file_limit.hpp : 32) Open file limit set to 1024 0:00:00.007 4M / 10M INFO General (kmer_splitters.hpp : 89) Memory available for splitting buffers: 2.66647 Gb 0:00:00.007 4M / 10M INFO General (kmer_splitters.hpp : 97) Using cell size of 4194304 0:00:00.007 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 97) Processing /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ1.fastq.gz 0:00:15.220 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 261324 reads 0:00:30.142 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 522582 reads 0:00:45.098 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 783660 reads 0:00:59.875 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 1044605 reads 0:01:14.664 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 1305536 reads 0:01:29.464 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 1566655 reads 0:02:13.894 646M / 646M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 2350067 reads 0:03:43.981 649M / 649M INFO K-mer Splitting (kmer_data.cpp : 97) Processing /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ2.fastq.gz 0:03:58.867 649M / 649M INFO K-mer Splitting (kmer_data.cpp : 107) Processed 4199506 reads 0:07:28.836 647M / 647M INFO K-mer Splitting (kmer_data.cpp : 97) Processing /media/aline/TOSHIBA EXT/Anous/clean-fastq/anous-66/split-adapter-quality-trimmed/anous-66-READ-singleton.fastq.gz 0:07:35.778 647M / 647M INFO K-mer Splitting (kmer_data.cpp : 112) Total 8003445 reads processed 0:07:35.807 7M / 552M INFO General (kmer_index_builder.hpp : 120) Starting k-mer counting. 0:08:06.660 7M / 552M INFO General (kmer_index_builder.hpp : 127) K-mer counting done. There are 104866056 kmers in total. 0:08:06.660 7M / 552M INFO General (kmer_index_builder.hpp : 133) Merging temporary buckets. 0:08:07.271 7M / 552M INFO K-mer Index Building (kmer_index_builder.hpp : 314) Building perfect hash indices 0:08:31.944 61M / 552M INFO General (kmer_index_builder.hpp : 150) Merging final buckets. 0:08:42.351 61M / 552M INFO K-mer Index Building (kmer_index_builder.hpp : 336) Index built. Total 48634696 bytes occupied (3.71023 bits per kmer). 0:08:42.414 61M / 552M INFO K-mer Counting (kmer_data.cpp : 351) Arranging kmers in hash map order 0:08:56.979 2G / 2G INFO General (main.cpp : 148) Clustering Hamming graph. 0:30:04.637 2G / 2G INFO General (main.cpp : 155) Extracting clusters 0:31:01.961 3G / 5G INFO General (main.cpp : 167) Clustering done. Total clusters: 56204771 0:31:01.988 2G / 5G INFO K-mer Counting (kmer_data.cpp : 371) Collecting K-mer information, this takes a while.

: Error in malloc(): out of memory Requested: 2516785344 == Error == system call for: "['/home/aline/.conda/envs/phyluce-1.7.1/bin/spades-hammer', '/home/aline/UCEs_Mari/spades/anous-66_spades/corrected/configs/config.info']" finished abnormally, OS return value: -6 In case you have troubles running SPAdes, you can write to spades.support@cab.spbu.ru or report an issue on our GitHub repository github.com/ablab/spades Please provide us with params.txt and spades.log files from the output directory. SPAdes log can be found here: /home/aline/UCEs_Mari/spades/anous-66_spades/spades.log Thank you for using SPAdes! 2022-11-10 10:44:33,753 - phyluce_assembly_assemblo_spades - INFO - Removing extraneous assembly files 2022-11-10 10:44:33,753 - phyluce_assembly_assemblo_spades - CRITICAL - Expected assembly files were not found in output. 2022-11-10 10:44:33,753 - phyluce_assembly_assemblo_spades - INFO - Symlinking assembled contigs into /home/aline/UCEs_Mari/spades/contigs 2022-11-10 10:44:33,754 - phyluce_assembly_assemblo_spades - INFO - =========== Completed phyluce_assembly_assemblo_spades ========== ________________________________________________________________________ I even tried to run only those two samples again, but I got the same error. Would you have any idea of what could have happened? Cheers,
brantfaircloth commented 2 years ago

This generally means that you either did not use or do not have sufficient RAM on the machine you are using to perform the assemblies. You can change the amount of RAM allocated to spades during the assembly process using the --memory argument to phyluce_assembly_assemblo_spades. By default, this value is set to 8 GB. If you have more ram on your machine, change this to the amount of RAM you have (or just below the amount. For example, if you have 16 GB, then:

phyluce_assembly_assemblo_spades \
--config <file> \
--cores <cores> \
--memory 15
marianamazzochi commented 2 years ago

Thanks, Branth. I will try to do that. But what could explain the fact that all of the other 65 files were processed with 8GB and only those last two were not?

brantfaircloth commented 2 years ago

It happens as a result of the amount of reads per sample and/or genome complexity (or lack thereof). Those differ by taxon (some taxa have more complex genomes) and individual (some individuals have more data than others).

marianamazzochi commented 2 years ago

These two samples have much less DNA than the others (~17 and 2ng/uL). As you said that lack of genome complexity could result in this error, do you think that the low concentration of DNA could have resulted in uselessness of those samples?

brantfaircloth commented 2 years ago

It's not DNA concentration - it's usually the number of reads generated by the sequence for a particular sample. And, it's usually more genome complexity, not less. Either way, increasing the RAM should fix the problem.

Jami5on commented 7 months ago

Hello @brantfaircloth !

I am having the same issue as above. After 30 or so successful samples, none of my remaining samples will run. I have since added "-- memory 100" and the job failed with the same issue. What am I missing? My next step is to run my samples with Spades outside of phyluce, but at this point, I would much rather understand why things aren't working before I do that.

Below is my shell script

!/bin/bash

SBATCH --job-name=UCE_assembly # Job name

SBATCH --mail-type=BEGIN,END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)

SBATCH --mail-user=jr4253@nau.edu # Where to send mail

SBATCH --nodes=1-1 # Run a single task

SBATCH --ntasks-per-node=16 # Number of CPU cores per task

SBATCH --mem=100gb # Job memory request

SBATCH --time=06-00:00:00 # Time limit days-hrs:min:sec

SBATCH --output=parallel_%j.log # Standard output and error log

SBATCH --chdir=/scratch/jr4253

module load anaconda3

conda activate phyluce-1.7.3

cd /scratch/jr4253

phyluce_assembly_assemblo_spades \ --conf spades3.conf \ --output spades-assemblies \ --cores 16 --memory 100

Attached is the log file for the run (saying it can't find the memory command)

Thank you in advance, Jami5on parallel_9703705.log

brantfaircloth commented 7 months ago

I can't speak to that. If the software is running for 30 samples, it is working as expected. A failure may be due to something about your system or your data, etc. You'll need to try spades outside of phyluce or look at the spades log file generated for the actual sample that is failing to determine what is occurring.

brantfaircloth commented 7 months ago

Also, the log file seems to show --memory 8 was used.