Memory usage with multithreading

hms-dbmi / scde

R package for analyzing single-cell RNA-seq data

http://pklab.med.harvard.edu/scde

Other

170 stars 64 forks source link

Memory usage with multithreading #34

Closed traversc closed 7 years ago

traversc commented 7 years ago

Hi Dr. Fan et. al.,

When we used SCDE on a dataset with multiple cores, we noticed there seemedd to be an issue with high memory usage. I think this is related to a long-standing bug in R parallelization here: https://github.com/tdhock/mclapply-memory

I modified the parallelization loop in scde's calculate.individual.models function: https://github.com/traversc/scde/commit/be34c81fb55bad46de178fac72904475f2d148bf

This seemed to reduce memory usage for us when run on 40 cores.

JEFworks commented 7 years ago

Hi Travers,

Thanks so much for the info and the modification. If you could please make a pull request, I can better test if your parallelization works with other platforms, and hopefully integrate your modifications.

Thanks again.

Best, Jean

traversc commented 7 years ago

It seems that there are no memory issues with the latest version on github, as we had issues with previous versions. Apologies for the redundancy!