search

stephenslab / fastTopics

Fast algorithms for fitting topic models and non-negative matrix factorizations to count data.
https://stephenslab.github.io/fastTopics
Other
77 stars 7 forks source link

Reduce memory usage in compute_lfc_stats_multicore #53

Open aksarkar opened 8 months ago

aksarkar commented 8 months ago

This change reduces memory usage by roughly 10%.

Profiling via mprof using the following snippet

devtools::load_all()
data(pbmc_facs)
temp <- de_analysis(pbmc_facs$fit, pbmc_facs$counts, verbose=FALSE, shrink.method='none', control=list(nc=32, nsplit=32))

on an AWS c7i.8xlarge instance yields the following usages.

On 295d7323 295d7323

On g19ec112 g19ec112:

pcarbo commented 8 months ago

@aksarkar I'm going to hold off on merging your PR for now. I think there is the potential to greatly improve the memory usage, but I have to play around with the implementation, and make sure that it doesn't also slow code the multithreading.

aksarkar commented 8 months ago

The timing of my test case indicates that it slightly improves the wallclock time also. In any case, it does not make appear to make the running time worse.