bonito duplex runs out of memory

[sathish-t]

Hello,

I use bonito 0.7.2 and the following bonito duplex command runs out of memory sometimes: bonito duplex --threads 30 $bam_file $pair_file > $duplex_basecalls_fq

While processing about 2.4 million reads on a slurm job, I got this error

> outputting unaligned fastq                                                                                         
> calling:   7%|##4                                   | 162658/2475888 [02:54<41:25, 930.64 pairs/s]slurmstepd: error
: Job 60123323 exceeded memory limit (144467984 > 125829120), being killed                    
slurmstepd: error: Exceeded job memory limit                                                                         
slurmstepd: error: *** JOB 60123323 ON t1024n1 CANCELLED AT 2024-05-22T20:27:36 ***    

But bonito ran just fine with a similar command processing about 2.9 million reads in a different job

Performing Bonito base-space duplex basecalling                                                                     
> outputting unaligned fastq                                                                                         
> completed reads: 2905869                                                                                           
> duration: 0:06:46                                                                                                  
> bases per second 8.7E+05                                                                                          
> done   

Any help is appreciated!

Regards Sathish

[sathish-t]

Hi,

I was able to solve the problem by using fewer threads (--threads 3). We are fine with this performance hit as there are other much slower steps in our pipeline. Do you have an inkling of why more threads are problematic?

Regards Sathish