PCA diagnostic error

[conchoecia]

I have a sparse dataframe with ~3 million columns of type int, and from 100-3000 samples. I pass a scipy lil dataframe with the values to umap for calculation. The whole process works fine, and I get mapper objects out with embeddings that make sense based on the data.

One problem I discovered was while trying the diagnostic tools. I got this error, for example, when trying the PCA diagnostic. I didn't find this error so far in the github issues, but perhaps it is related to: https://github.com/lmcinnes/umap/pull/911

The core error is this: TypeError: PCA only support sparse inputs with the "arpack" solver, while "auto" was passed. See TruncatedSVD for a possible alternative.

This is how I generate the objects from the lil matrix: with my parameters n and min_dist:

reducer = umap.UMAP(low_memory=True, n_neighbors = n, min_dist = min_dist)
mapper = reducer.fit(lil)

This was the traceback from where I made the call in my program:

    umap.plot.diagnostic(mapper, diagnostic_type='pca')
  File "/lisc/user/schultz/miniconda3/lib/python3.11/site-packages/umap/plot.py", line 1055, in diagnostic
    color_proj = sklearn.decomposition.PCA(n_components=3).fit_transform(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lisc/user/schultz/miniconda3/lib/python3.11/site-packages/sklearn/utils/_set_output.py", line 295, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lisc/user/schultz/miniconda3/lib/python3.11/site-packages/sklearn/base.py", line 1474, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lisc/user/schultz/miniconda3/lib/python3.11/site-packages/sklearn/decomposition/_pca.py", line 454, in fit_transform
    U, S, Vt = self._fit(X)
               ^^^^^^^^^^^^
  File "/lisc/user/schultz/miniconda3/lib/python3.11/site-packages/sklearn/decomposition/_pca.py", line 472, in _fit
    raise TypeError(
TypeError: PCA only support sparse inputs with the "arpack" solver, while "auto" was passed. See TruncatedSVD for a possible alternative.

Attached is a screenshot of a bokeh instance with my results from the lil matrix with ~3000 samples, colors are from my own annotation of known groups in the data:

[lmcinnes]

Hmm, I think the short answer is that the PCA diagnostic plot won't work for sparse data -- I don't think there are any tests for that particular combination. This ultimately comes down to the sklearn PCA implementation, so it's not so easy to fix at this end. Sorry.

[conchoecia]

Ok, thank you for your thoughts!