Closed Intron7 closed 1 year ago
You are likely running into the error of CUSPARSE_STATUS_INSUFFICIENT_RESOURCES
(see here).
On my 48GB GPU I was not able to run the sparse PCA with a density of 0.05
, but by lowering the density to less than 0.02
I was able to run it.
By quickly running a few sparse PCA I saw that for a matrix of that dimension, a density of 0.02 leads to a 26GB memory allocation. This is related to the covariance matrix computation. https://github.com/rapidsai/cuml/blob/93f7ddc6c473d93093900061afd539c08b53287f/python/cuml/decomposition/pca.pyx#L365
So you can either lower the dimension of the matrix or lower it's density so that it fits your GPU memory.
Ok thank you. I'm just confused because the dense version works with way bigger matrices. Usually sparse matrices are used to save VRAM.
@lowener why was this closed? Doesn’t it seem like a bug that dense PCA needs less memory than sparse PCA?
I think it might be a bug with cuSPARSE. I created a replacement implementation for the cov
function with a custom cupy kernel for csr matrices based on a spGEMM paper. It performs really well. I can create a PR.
@Intron7 This would be a great new feature that would be very helpful for many users! Your PR would be very welcomed! CuSparse has recently released more algorithm (as of this February) for SpGEMM that are using less memory. (see here) So I created this issue https://github.com/cupy/cupy/issues/7699 to add them in cupy. @flying-sheep I closed this issue too early. I'll reopen it.
Describe the bug The PCA function doesn't work with sparse matrices. Steps/Code to reproduce bug
Output
Expected behavior To run PCA with sparse matrix without the error and get the same result.
Environment details (please complete the following information):
Click here to see environment details
Additional context Has also been prevalent in older rapids versions.