search

farrellja / URD

URD - Reconstruction of Branching Developmental Trajectories
GNU General Public License v3.0
115 stars 41 forks source link

memory blowout on createURD #56

Open decarlin opened 4 years ago

decarlin commented 4 years ago

I'm looking at a largish dataset, 190k cells, coming from a Seurat object. My data matrix is coming in spase as a dgCMatrix but when I go to make an URD object i get the following error:

axial <- createURD(count.data = counts, meta = meta)

2020-04-17 16:28:00: Filtering cells by number of genes. Error in asMethod(object) : Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105

This is a big memory (~700 GB) Unix machine, and it seems like it should all work, given that in the past I have had little problem running 10k cells on my 16gb mac. I'm wondering if the matrix is accidentally loosing its sparseness in the filtering call?

farrellja commented 4 years ago

@decarlin It turns out that this is because it turns out that apply silently converts sparse matrices back to dense ones. I believe that I've corrected it, and I'm emailing you a new version of the function to try.

farrellja commented 4 years ago

@decarlin createURD and findVariableGenes have been updated in the debug branch (devtools::install_github("farrellja/URD@debug")). I did a search and don't see any additional calls to apply or sweep on the full @count.data or @logupx.data matrices, so hopefully things should run smoothly now. Let me know how it goes with your large data with those updates.