Closed daniel-spies closed 9 months ago
Hi Daniel,
Thanks for your interest in our work & thank you for brining this issue to our attention!
This is likely due to the fact that our previous implementation was designed to take count matrix in the format of a numpy array, and matrix multiplication was implemented with numpy functions. However, np.dot behaves unexpectedly when the count matrix is in the format of scipy csr_matrix. It generates a large number of p*p matrices, instead of computing a dot product, which may explain the memory issue.
We have updated the implementation to take AnnData / csr_matrix as input. You can follow this notebook for an example, or CSCORE_IRLS.py for the actual implementations. We have also benchmarked the time and memory usage of this new implementation at this notebook. This implementation should give comparable speed as the R version.
I hope this helps! Feel free to leave a comment if you have more questions.
Best, Chang
Hi there, thanks for the great tool.
I followed the python IRLS example but after a short while I get an out of memory error and python is killed (working on a cluster). Even requesting up to 512GB of RAM did not help. I used a cluster subset of 500 cells with 3000 genes, so this should not be a problem.
When running the same count table imported into R the results are there within a few seconds, somehow you python implementation is killing itself. I'm running CS-CORE with the following package versions:
python 3.9.16 numpy 1.23.5 scipy 1.7.2
Looking forward to use it in the future!
best Daniel