error in sobolev_alignment_clf.krr_model_selection()

[joseph-siefert]

Report

Hi Soufiane, the scVI training runs no problem, but the KRR gives the following error:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[27], line 3
      1 #KRR approximation
      2 # Select best model for KRR
----> 3 sobolev_alignment_clf.krr_model_selection(
      4     X_source=source_an,
      5     X_target=target_an,
      6     M=500,
      7     same_model_alignment_thresh=0.95
      8 )

File /mambaforge/envs/sobolev/lib/python3.9/site-packages/sobolev_alignment/sobolev_alignment.py:1018, in SobolevAlignment.krr_model_selection(self, X_source, X_target, M, same_model_alignment_thresh)
   1016 X_input = {"source": X_source.X, "target": X_target.X}
   1017 if self._fit_params["log_input"]:
-> 1018     X_input = {k: np.log10(X_input[k] + 1) for k in X_input}
   1020 # Compute sigma after re-scaling data (if required)
   1021 if self._fit_params["frob_norm_source"]:

File /mambaforge/envs/sobolev/lib/python3.9/site-packages/sobolev_alignment/sobolev_alignment.py:1018, in <dictcomp>(.0)
   1016 X_input = {"source": X_source.X, "target": X_target.X}
   1017 if self._fit_params["log_input"]:
-> 1018     X_input = {k: np.log10(X_input[k] + 1) for k in X_input}
   1020 # Compute sigma after re-scaling data (if required)
   1021 if self._fit_params["frob_norm_source"]:

File /mambaforge/envs/sobolev/lib/python3.9/site-packages/scipy/sparse/_base.py:462, in _spbase.__add__(self, other)
    460         return self.copy()
    461     # Now we would add this scalar to every element.
--> 462     raise NotImplementedError('adding a nonzero scalar to a '
    463                               'sparse array is not supported')
    464 elif issparse(other):
    465     if other.shape != self.shape:

NotImplementedError: adding a nonzero scalar to a sparse array is not supported

There are raw counts in both .X and .layers['counts'] for both source and target

Version information

-----
anndata             0.9.2
numpy               1.24.4
pandas              2.0.3
scanpy              1.9.3
scvi                1.0.3
session_info        1.0.0
sobolev_alignment   0.4.1
torch               2.0.0
tqdm                4.66.1
-----
Cython              3.0.0
PIL                 9.4.0
absl                NA
aiohttp             3.8.5
aiosignal           1.3.1
annotated_types     0.5.0
anyio               NA
arrow               1.2.3
asttokens           NA
async_timeout       4.0.3
attr                23.1.0
babel               2.12.1
backcall            0.2.0
backoff             2.2.1
brotli              1.0.9
bs4                 4.12.2
certifi             2023.07.22
cffi                1.15.1
charset_normalizer  3.2.0
chex                0.1.7
click               8.1.7
cloudpickle         2.2.1
colorama            0.4.6
comm                0.1.4
contextlib2         NA
croniter            NA
cycler              0.10.0
cython              3.0.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.8
decorator           5.1.1
deepdiff            6.3.1
defusedxml          0.7.1
docrep              0.3.2
entrypoints         0.4
executing           1.2.0
falkon              0.8.3
fastapi             0.101.1
fastjsonschema      NA
flax                0.6.1
fqdn                NA
frozenlist          1.4.0
fsspec              2023.6.0
future              0.18.3
gmpy2               2.1.2
h5py                3.9.0
hyperopt            0.2.7
idna                3.4
igraph              0.10.6
importlib_metadata  NA
importlib_resources NA
ipykernel           6.25.1
ipython_genutils    0.2.0
ipywidgets          8.1.0
isoduration         NA
jax                 0.4.14
jaxlib              0.4.14
jedi                0.19.0
jinja2              3.1.2
joblib              1.3.2
json5               NA
jsonpointer         2.0
jsonschema          4.17.3
jupyter_events      0.6.3
jupyter_server      2.7.1
jupyterlab_server   2.24.0
keopscore           2.1.2
kiwisolver          1.4.4
leidenalg           0.10.1
lightning           2.0.7
lightning_cloud     NA
lightning_fabric    2.0.7
lightning_utilities 0.9.0
llvmlite            0.40.1
markupsafe          2.1.3
matplotlib          3.7.2
ml_collections      NA
ml_dtypes           0.2.0
mpl_toolkits        NA
mpmath              1.3.0
msgpack             1.0.5
mudata              0.2.3
multidict           6.0.4
multipart           0.0.6
multipledispatch    0.6.0
natsort             8.4.0
nbformat            5.9.2
networkx            3.1
numba               0.57.1
numpyro             0.13.0
nvfuser             NA
opt_einsum          v3.3.0
optax               0.1.7
ordered_set         4.1.0
orjson              3.9.5
overrides           NA
packaging           23.1
parso               0.8.3
past                0.18.3
patsy               0.5.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.10.0
prometheus_client   NA
prompt_toolkit      3.0.39
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pvectorc            NA
py4j                0.10.9.7
pydantic            2.1.1
pydantic_core       2.4.0
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.16.1
pykeops             2.1.2
pyparsing           3.0.9
pyro                1.8.6+4be5c2e
pyrsistent          NA
pythonjsonlogger    NA
pytorch_lightning   2.0.7
pytz                2023.3
requests            2.31.0
rfc3339_validator   0.1.4
rfc3986_validator   0.1.1
rich                NA
scipy               1.11.2
seaborn             0.12.2
send2trash          NA
setuptools          68.1.2
six                 1.16.0
sklearn             1.3.0
sniffio             1.3.0
socks               1.7.1
soupsieve           2.3.2.post1
sparse              0.14.0
stack_data          0.6.2
starlette           0.27.0
statsmodels         0.14.0
sympy               1.12
texttable           1.6.7
threadpoolctl       3.2.0
tomli               2.0.1
toolz               0.12.0
torchaudio          2.0.0
torchmetrics        1.0.3
torchvision         0.15.0
tornado             6.3.3
traitlets           5.9.0
tree                0.1.8
typing_extensions   NA
uri_template        NA
urllib3             2.0.4
uvicorn             0.23.2
wcwidth             0.2.6
webcolors           1.13
websocket           1.6.2
websockets          11.0.3
xarray              2023.8.0
yaml                6.0.1
yarl                1.9.2
zipp                NA
zmq                 24.0.1
zoneinfo            NA
-----
IPython             8.14.0
jupyter_client      7.4.9
jupyter_core        5.3.1
jupyterlab          3.6.5
notebook            6.5.5
-----
Python 3.9.17 | packaged by conda-forge | (main, Aug 10 2023, 07:02:31) [GCC 12.3.0]
Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17

[saroudant]

Hi Joe,

Thanks a lot for raising this issue.
It seems that there is an issue with the scipy sparse matrix structure. Could you please tell me what happens if you "densify" the matrix prior to running Sobolev Alignment, e.g.,:

data_an.X = np.array(data_an.X.todense())

If it resolves the problem, I can add it as a pre-processing step within the fit procedure.

Have a great day, Souf

[joseph-siefert]

Thanks Soufiane, this did get around that error. Now I have a different issue with Falkon. I have Pytorch 2.0.0 and CUDA 11.7, and I can access GPU and run scVI no problem. I have compiled Falkon with the wheel for these specific versions, which should be tested and supported but is not working. I am following up with them on it.

[saroudant]

Thanks a lot. I will add that in the pre-processing pipeline, and in the tutorial.

For Falkon, it took me quite some trial-and-error to install it successfully. I gave some advice here: https://github.com/NKI-CCB/sobolev_alignment#issues-with-the-compiler I am not sure that cuda version plays a role here, but pytorch should be at 1.11 (2.0 is very different). The compiler must also be chosen differently. I'll follow the correspondence you have with them and add it to the README.

[joseph-siefert]

Falkon is working now, but I came across another error further down the line:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[61], line 2
      1 #Estimate number of shared principal vectors
----> 2 n_similar_pvs = sobolev_alignment_clf.permutation_test_number_similar_pvs(n_permutations=10)

AttributeError: 'SobolevAlignment' object has no attribute 'permutation_test_number_similar_pvs'

[saroudant]

Hi Joe,

Great that the FalkonML team got it sorted so fast. For this new issue, it seems that you have version 0.4.1, while the last release is 1.0.0. I fixed this bug in the last release normally. You can install it using pip install sobolev-alignment command. You do not need to install Falkon again, as long as you re-install in the same environment. Let me know if the problem remains.

Best, Soufiane

[joseph-siefert]

I installed through pip, and running the command gives:

Requirement already satisfied: sobolev-alignment in /mambaforge/envs/sobolev/lib/python3.9/site-packages (1.0.0)

yet:

tail -1 /mambaforge/envs/sobolev/lib/python3.9/site-packages/sobolev_alignment/__init__.py    
__version__ = "0.4.1"

I tried to force reinstall:

pip install --upgrade --no-deps --force-reinstall sobolev-alignment
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting sobolev-alignment
  Downloading sobolev_alignment-1.0.0-py3-none-any.whl (34 kB)
Installing collected packages: sobolev-alignment
  Attempting uninstall: sobolev-alignment
    Found existing installation: sobolev_alignment 1.0.0
    Uninstalling sobolev_alignment-1.0.0:
      Successfully uninstalled sobolev_alignment-1.0.0
Successfully installed sobolev-alignment-1.0.0

but still:

tail -1 /mambaforge/envs/sobolev/lib/python3.9/site-packages/sobolev_alignment/__init__.py    
__version__ = "0.4.1"