search

PacificBiosciences / pbbioconda

PacBio Secondary Analysis Tools on Bioconda. Contains list of PacBio packages available via conda.
BSD 3-Clause Clear License
251 stars 45 forks source link

Auto thread detection in pbmm2 too high on slurm HPC #707

Open mrvollger opened 2 months ago

mrvollger commented 2 months ago

Operating system NAME="Rocky Linux" VERSION="8.9 (Green Obsidian)"

Package name pbmm2 v1.13.0

issue When I run pbmm2 I am finding that pbmm2 detects the total number of cores on the machine rather than what I have access to via my job on a slurm HPC. For example, here, pbmm2 detects 192 cores, but I only have 64, really:

$ pbmm2 align -m 16g --sort --unmapped denovo-asm/GM12878.v0.1.0.scaffold.fa  ../../k-mer-variant-phasing/results/GM12878/GM12878.haplotagged.bam  tmp.bam --log-level INFO
>|> 20240816 19:19:00.715 -|- INFO -|- AlignSettings -|- 0x7f68af197140|| -|- Using 184 threads for alignments, 8 threads for sorting, and 128G bytes RAM for sorting.

$ nproc
64

Thanks, Mitchell

armintoepfer commented 2 months ago

Then it sounds like slurm won't properly allocate and restrict number of CPUs to your job. There are several ways to adjust the number of threads, for example using -j

mrvollger commented 2 months ago

Happy to use -j, but just in case I wasn't clear, the node can only use 64 cores, so when pbmm2 detects 192, its CPU utilization is still only going to those 64 cores. Mostly, I was pointing out that there is some detection difference between e.g. nproc and pbmm2 that allows nproc to detect slurm's limit of 64 in this example.

armintoepfer commented 1 month ago

I've created an internal ticket to look at this issue.