search

CRC-FONDA / A2-metagenome-snakemake

Collection of metagenomic read mapping workflows that have been tailored for different computational architectures.
0 stars 0 forks source link

Forcing threads instead of processes #7

Closed eaasna closed 3 years ago

eaasna commented 3 years ago

Running the low memory prefilter binary on all_reads.fastq shows that the program is able to use 8 threads in parallel.

When asking snakemake to run 64 bins of reads with 8 cores it has not been possible to force each bin to be processed using 8 threads. Instead 8 processes are created. Each of which reads the hash table into memory. This results in memory overload.

The same procedure seems to work fine using DREAM tools, even though the IBF is 1.1G where the hash table just 93M.

eaasna commented 3 years ago

The bug was actually with the hashmap filter. There were multiple threads accessing the same 64bit word and this led to undefined behavior