Closed jberkh closed 1 month ago
Thanks! I'll take a look at this
Just to give a quick update on this, the simplest patch is to explicitly convert the unspliced data into a dense array when it comes in as sparse since there's no argsort
equivalent for csr_matrix
. I don't think this is the best solution because it will require more memory than necessary.
I figured as much, given that np.argsort by defnition returns a dense matrix since all the zero value return non-zero indices. Unfortunately, calling .todense() on too large datasets does blow up in memory quite a bit, possibly to the point of crashing.
Maybe solution could be to convert the matrices to dense in batches, something along these lines:
if isinstance(unspliced, csr_matrix) or isinstance(unspliced, csr_matrix):
sorted_unspliced = np.empty(unspliced.shape)
# batch size of 1000 here
for i in range(0, unspliced_shape.shape[1], 1000):
sorted[:, i:i+1000] = np.argsort(unspliced[:, i:i+1000].todense(), axis = 0)
Thanks for the suggestion. I'm personally leaning towards a solution that natively deals with the sparse matrix instead of having to implicitly convert to dense. I'm also looking into what might be a potential bug in this whole preprocessing step.
I’m sorry for the long time it took us to get back. We expect users to preprocess data using: https://scvelo.readthedocs.io/en/stable/scvelo.pp.moments.html as highlighted in our tutorial. This function returns dense matrices. Please follow our tutorial. It should be safe to downsample n_cells after running the moments function, reduce highly variable genes to less genes or focus your analysis on several celltypes to overcome memory constraints.
Issue
The
scvi.external.velovi.VELOVI
class usesnp.argsort
on the spliced/unspliced matrices, but this returnsarray([0])
when these matrices are sparse.Minimally reproducible code
Versions:
(Installed with
pip install git+https://github.com/scverse/scvi-tools
on 4/4/24 ~12:00, as velovi is not available in the current pypi/conda release)