R killed when using parallelization

SingleR-inc / SingleR

Clone of the Bioconductor repository for the SingleR package.

https://bioconductor.org/packages/devel/bioc/html/SingleR.html

GNU General Public License v3.0

177 stars 19 forks source link

R killed when using parallelization #214

Closed ycl6 closed 2 years ago

ycl6 commented 2 years ago

Hi,

I was trying parallelization (BiocParallel::MulticoreParam) when running SingleR, but ran into similar problem I encountered when using runPCA from scater (https://github.com/Alanocallaghan/scater/issues/148) and emptyDrops from DropletUtils (https://github.com/MarioniLab/DropletUtils/issues/65), i.e. it spawned multiple threads but they never complete and killed R eventually. I am wondering could the same changes also be a fix for SingleR?

LTLA commented 2 years ago

I don't think so. SingleR's configuration is correct in this respect - unlike the other two cases you mentioned, SingleR never used .bpNotSharedOrUp. My best guess is that you're running out of memory; the multicore parallelization paradigm is not very smart about how memory is shared across workers, which causes particular problems on HPCs with hard memory limits.

ycl6 commented 2 years ago

@LTLA Thanks for the comment, I think you are right about running out of memory, I'll reduce the number of workers next time.