Open mr-eyes opened 4 years ago
I suspect the issue to be in Python memory management and deallocation for the flat_parallel_hashmap holding the kmers chunks. To be followed ...
conda install -c bioconda -y bcalm
conda install -c bioconda -y sra-tools
wget -c https://sra-download.ncbi.nlm.nih.gov/traces/sra51/SRR/010757/SRR11015356 -O SRR11015356.sra
fastq-dump --fasta 0 --split-files SRR11015356.sra
ls -1 *fasta > list_reads
bcalm -kmer-size 75 -max-memory 12000 -out SRR11015356_k75 -in list_reads
file format type num_seqs sum_len min_len avg_len max_len
SRR11015356_k75.unitigs.fa FASTA DNA 11,824,622 1,133,741,131 75 95.9 1,683
grep ">" SRR11015356_k75.unitigs.fa | cut -c2- | awk -F' ' '{print $0"\t"$1}' > SRR11015356_k75.unitigs.fa.names
import kProcessor as kp
fasta_file = "SRR11015356_k75.unitigs.fa"
names_file = fasta_file + ".names"
kSize, Q, mode, chunk_size = 25, 29, 1, int(1e4)
kf = kp.kDataFrameMQF(kSize, Q, mode)
print("Indexing ...")
ckf = kp.index(kf, {"mode" : mode}, fasta_file , chunk_size, names_file)
print("Serializing the index ...")
ckf.save("idx_k25_SRR11015356_k75.unitigs")
After having some debugging in the C++ source code, now it's certain that the huge size of hashmap legends is the bottleneck.
The huge number of colors causes slowness in insertion and inflation in the memory consumption
Reproduce the issue
1. Required packages
conda install -c bioconda -y bcalm
sra-tools
conda install -c bioconda -y sra-tools
2. Data preparation
3. Creating cDBG
3.1 cDBG creation with k=75
3.2 cDBG unitigs fasta file stats
4. kProcessor indexing
4.1 Generating names file
4.2 Indexing
4.3 Output
Click to the expand the output!
```bash stack trace: 0 [0x7f3c4f5f7c12] /home/mabuelanin/miniconda3/envs/kspider/lib/python3.7/site-packages/_kProcessor.cpython-37m-x86_64-linux-gnu.so(+0x356c12) 1 [0x7f3c50f32f20] /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) 2 [0x7f3c51082b53] /lib/x86_64-linux-gnu/libc.so.6(+0x18eb53) 3 [0x7f3c4f6062ad] std::vector