saezlab / decoupler-py

Python package to perform enrichment analysis from omics data.
https://decoupler-py.readthedocs.io/
GNU General Public License v3.0
154 stars 23 forks source link

AttributeError: 'csr_matrix' object has no attribute 'A' #139

Closed MarwaneBourdim closed 2 months ago

MarwaneBourdim commented 2 months ago

Hi,

When I try to reproduce the pathway activity inference vignette, I try to run:

dc.run_mlm(
    mat=adata,
    net=progeny,
    source='source',
    target='target',
    weight='weight',
    verbose=True
)

And I get the following Attribute error:

/decoupler/method_mlm.py:58, in mlm(mat, net, batch_size, verbose)
     54 for i in tqdm(range(n_batches), disable=not verbose):
     55 
     56     # Subset batch
     57     srt, end = i * batch_size, i * batch_size + batch_size
---> 58     y = mat[srt:end].A.T
     60     # Compute MLM for batch
     61     es[srt:end] = fit_mlm(net, y, inv, df)[:, 1:]

AttributeError: 'csr_matrix' object has no attribute 'A'

Looking at the code, it makes sense to me since adata.raw.X is not of type np.matrix and doesn't have attribute "A". So I've tried to convert it with:

adata.X = np.matrix(adata.X)
adata.raw = adata

But when I do that I get this other error:

TypeError: matrix.sum() got an unexpected keyword argument 'keepdims'

In the meantime to circumvent it, I simply convert adata.raw to a pandas Dataframe like that :

mat_ = adata.raw.to_adata().to_df()
adata.obsm["mlm_estimate"], adata.obsm["mlm_pvals"] = dc.run_mlm(
    mat=mat_,
    net=progeny,
    source='source',
    target='target',
    weight='weight',
    verbose=True
)

I think it works this way, but I'd like to know what's wrong with the other way.

This problem happens in the following environment:

  _libgcc_mutex                      0.1           conda_forge              conda-forge
  _openmp_mutex                      4.5           2_gnu                    conda-forge
  _r-mutex                           1.0.1         anacondar_1              conda-forge
  anndata                            0.10.8        pyhd8ed1ab_0             conda-forge
  anndata2ri                         1.3.1         pyhdfd78af_1             bioconda   
  argcomplete                        3.4.0         pyhd8ed1ab_0             conda-forge
  array-api-compat                   1.7.1         pyhd8ed1ab_0             conda-forge
  asttokens                          2.4.1         pyhd8ed1ab_0             conda-forge
  binutils_impl_linux-64             2.40          ha1999f0_7               conda-forge
  brotli                             1.1.0         hd590300_1               conda-forge
  brotli-bin                         1.1.0         hd590300_1               conda-forge
  bwidget                            1.9.14        ha770c72_1               conda-forge
  bzip2                              1.0.8         hd590300_5               conda-forge
  c-ares                             1.28.1        hd590300_0               conda-forge
  ca-certificates                    2024.6.2      hbcca054_0               conda-forge
  cached-property                    1.5.2         hd8ed1ab_1               conda-forge
  cached_property                    1.5.2         pyha770c72_1             conda-forge
  cairo                              1.18.0        hbb29018_2               conda-forge
  certifi                            2024.6.2      pyhd8ed1ab_0             conda-forge
  cffi                               1.16.0        py311hb3a22ac_0          conda-forge
  colorama                           0.4.6         pyhd8ed1ab_0             conda-forge
  comm                               0.2.2         pyhd8ed1ab_0             conda-forge
  contourpy                          1.2.1         py311h9547e67_0          conda-forge
  curl                               8.8.0         he654da7_1               conda-forge
  cycler                             0.12.1        pyhd8ed1ab_0             conda-forge
  debugpy                            1.8.2         py311h4332511_0          conda-forge
  decorator                          5.1.1         pyhd8ed1ab_0             conda-forge
  dunamai                            1.21.2        pyhd8ed1ab_0             conda-forge
  exceptiongroup                     1.2.0         pyhd8ed1ab_2             conda-forge
  executing                          2.0.1         pyhd8ed1ab_0             conda-forge
  expat                              2.6.2         h59595ed_0               conda-forge
  font-ttf-dejavu-sans-mono          2.37          hab24e00_0               conda-forge
  font-ttf-inconsolata               3.000         h77eed37_0               conda-forge
  font-ttf-source-code-pro           2.038         h77eed37_0               conda-forge
  font-ttf-ubuntu                    0.83          h77eed37_2               conda-forge
  fontconfig                         2.14.2        h14ed4e7_0               conda-forge
  fonts-conda-ecosystem              1             0                        conda-forge
  fonts-conda-forge                  1             0                        conda-forge
  fonttools                          4.53.0        py311h331c9d8_0          conda-forge
  freetype                           2.12.1        h267a509_2               conda-forge
  fribidi                            1.0.10        h36c2ea0_0               conda-forge
  gcc_impl_linux-64                  14.1.0        h3c94d91_0               conda-forge
  get-annotations                    0.1.2         pyhd8ed1ab_0             conda-forge
  get_version                        3.5.5         pyhd8ed1ab_0             conda-forge
  gfortran_impl_linux-64             14.1.0        he4a1faa_0               conda-forge
  git                                2.45.2        pl5321ha099dd3_1         conda-forge
  graphite2                          1.3.13        h59595ed_1003            conda-forge
  gsl                                2.7           he838d99_0               conda-forge
  gxx_impl_linux-64                  14.1.0        h2879b86_0               conda-forge
  h5py                               3.11.0        nompi_py311h439e445_102  conda-forge
  harfbuzz                           9.0.0         hfac3d4d_0               conda-forge
  hdf5                               1.14.3        nompi_hdf9ad27_105       conda-forge
  icu                                73.2          h59595ed_0               conda-forge
  importlib-metadata                 8.0.0         pyha770c72_0             conda-forge
  importlib_metadata                 8.0.0         hd8ed1ab_0               conda-forge
  ipykernel                          6.29.5        pyh3099207_0             conda-forge
  ipython                            8.26.0        pyh707e725_0             conda-forge
  jedi                               0.19.1        pyhd8ed1ab_0             conda-forge
  jinja2                             3.1.4         pyhd8ed1ab_0             conda-forge
  joblib                             1.4.2         pyhd8ed1ab_0             conda-forge
  jq                                 1.7.1         hd590300_0               conda-forge
  jupyter_client                     8.6.2         pyhd8ed1ab_0             conda-forge
  jupyter_core                       5.7.2         py311h38be061_0          conda-forge
  kernel-headers_linux-64            2.6.32        he073ed8_17              conda-forge
  keyutils                           1.6.1         h166bdaf_0               conda-forge
  kiwisolver                         1.4.5         py311h9547e67_1          conda-forge
  krb5                               1.21.3        h659f571_0               conda-forge
  lcms2                              2.16          hb7c19ff_0               conda-forge
  ld_impl_linux-64                   2.40          hf3520f5_7               conda-forge
  legacy-api-wrap                    1.4           pyhd8ed1ab_1             conda-forge
  lerc                               4.0.0         h27087fc_0               conda-forge
  libaec                             1.1.3         h59595ed_0               conda-forge
  libblas                            3.9.0         22_linux64_openblas      conda-forge
  libbrotlicommon                    1.1.0         hd590300_1               conda-forge
  libbrotlidec                       1.1.0         hd590300_1               conda-forge
  libbrotlienc                       1.1.0         hd590300_1               conda-forge
  libcblas                           3.9.0         22_linux64_openblas      conda-forge
  libcurl                            8.8.0         hca28451_1               conda-forge
  libdeflate                         1.20          hd590300_0               conda-forge
  libedit                            3.1.20191231  he28a2e2_2               conda-forge
  libev                              4.33          hd590300_2               conda-forge
  libexpat                           2.6.2         h59595ed_0               conda-forge
  libffi                             3.4.2         h7f98852_5               conda-forge
  libgcc-devel_linux-64              14.1.0        h5d3d1c9_100             conda-forge
  libgcc-ng                          14.1.0        h77fa898_0               conda-forge
  libgfortran-ng                     14.1.0        h69a702a_0               conda-forge
  libgfortran5                       14.1.0        hc5f4f2c_0               conda-forge
  libglib                            2.80.2        h8a4344b_1               conda-forge
  libgomp                            14.1.0        h77fa898_0               conda-forge
  libhwloc                           2.10.0        default_h5622ce7_1001    conda-forge
  libiconv                           1.17          hd590300_2               conda-forge
  libjpeg-turbo                      3.0.0         hd590300_1               conda-forge
  liblapack                          3.9.0         22_linux64_openblas      conda-forge
  libllvm14                          14.0.6        hcd5def8_4               conda-forge
  libnghttp2                         1.58.0        h47da74e_1               conda-forge
  libnsl                             2.0.1         hd590300_0               conda-forge
  libopenblas                        0.3.27        pthreads_h413a1c8_0      conda-forge
  libpng                             1.6.43        h2797004_0               conda-forge
  libsanitizer                       14.1.0        hcba0ae0_0               conda-forge
  libsodium                          1.0.18        h36c2ea0_1               conda-forge
  libsqlite                          3.46.0        hde9e2c9_0               conda-forge
  libssh2                            1.11.0        h0841786_0               conda-forge
  libstdcxx-devel_linux-64           14.1.0        h5d3d1c9_100             conda-forge
  libstdcxx-ng                       14.1.0        hc0a3c3a_0               conda-forge
  libtiff                            4.6.0         h1dd3fc0_3               conda-forge
  libuuid                            2.38.1        h0b41bf4_0               conda-forge
  libwebp-base                       1.4.0         hd590300_0               conda-forge
  libxcb                             1.16          hd590300_0               conda-forge
  libxcrypt                          4.4.36        hd590300_1               conda-forge
  libxml2                            2.12.7        hc051c1a_1               conda-forge
  libzlib                            1.3.1         h4ab18f5_1               conda-forge
  llvmlite                           0.43.0        py311hbde99c3_0          conda-forge
  make                               4.3           hd18ef5c_1               conda-forge
  markupsafe                         2.1.5         py311h459d7ec_0          conda-forge
  matplotlib-base                    3.8.4         py311ha4ca890_2          conda-forge
  matplotlib-inline                  0.1.7         pyhd8ed1ab_0             conda-forge
  munkres                            1.1.4         pyh9f0ad1d_0             conda-forge
  natsort                            8.4.0         pyhd8ed1ab_0             conda-forge
  ncurses                            6.5           h59595ed_0               conda-forge
  nest-asyncio                       1.6.0         pyhd8ed1ab_0             conda-forge
  networkx                           3.3           pyhd8ed1ab_1             conda-forge
  numba                              0.60.0        py311h4bc866e_0          conda-forge
  numpy                              1.26.4        py311h64a7726_0          conda-forge
  oniguruma                          6.9.9         hd590300_0               conda-forge
  openjpeg                           2.5.2         h488ebb8_0               conda-forge
  openssl                            3.3.1         h4ab18f5_1               conda-forge
  packaging                          24.1          pyhd8ed1ab_0             conda-forge
  pandas                             2.2.2         py311h14de704_1          conda-forge
  pango                              1.54.0        h4c5309f_1               conda-forge
  parso                              0.8.4         pyhd8ed1ab_0             conda-forge
  patsy                              0.5.6         pyhd8ed1ab_0             conda-forge
  pcre2                              10.44         h0f59acf_0               conda-forge
  perl                               5.32.1        7_hd590300_perl5         conda-forge
  pexpect                            4.9.0         pyhd8ed1ab_0             conda-forge
  pickleshare                        0.7.5         py_1003                  conda-forge
  pillow                             10.3.0        py311h82a398c_1          conda-forge
  pip                                24.0          pyhd8ed1ab_0             conda-forge
  pixman                             0.43.2        h59595ed_0               conda-forge
  platformdirs                       4.2.2         pyhd8ed1ab_0             conda-forge
  prompt-toolkit                     3.0.47        pyha770c72_0             conda-forge
  psutil                             6.0.0         py311h331c9d8_0          conda-forge
  pthread-stubs                      0.4           h36c2ea0_1001            conda-forge
  ptyprocess                         0.7.0         pyhd3deb0d_0             conda-forge
  pure_eval                          0.2.2         pyhd8ed1ab_0             conda-forge
  pycparser                          2.22          pyhd8ed1ab_0             conda-forge
  pygments                           2.18.0        pyhd8ed1ab_0             conda-forge
  pynndescent                        0.5.13        pyhff2d567_0             conda-forge
  pyparsing                          3.1.2         pyhd8ed1ab_0             conda-forge
  python                             3.11.9        hb806964_0_cpython       conda-forge
  python-dateutil                    2.9.0         pyhd8ed1ab_0             conda-forge
  python-tzdata                      2024.1        pyhd8ed1ab_0             conda-forge
  python_abi                         3.11          4_cp311                  conda-forge
  pytz                               2024.1        pyhd8ed1ab_0             conda-forge
  pyyaml                             6.0.1         py311h459d7ec_1          conda-forge
  pyzmq                              26.0.3        py311h08a0b41_0          conda-forge
  readline                           8.2           h8228510_1               conda-forge
  rpy2                               3.5.11        py311r43h1f0f07a_3       conda-forge
  scanpy                             1.10.2        pyhd8ed1ab_0             conda-forge
  scikit-learn                       1.5.0         py311he08f58d_1          conda-forge
  scikit-misc                        0.1.4         py311ha9cf758_4          conda-forge
  scipy                              1.14.0        py311h517d4fd_0          conda-forge
  seaborn                            0.13.2        hd8ed1ab_2               conda-forge
  seaborn-base                       0.13.2        pyhd8ed1ab_2             conda-forge
  sed                                4.8           he412f7d_0               conda-forge
  session-info                       1.0.0         pyhd8ed1ab_0             conda-forge
  setuptools                         70.1.1        pyhd8ed1ab_0             conda-forge
  simplegeneric                      0.8.1         py_1                     conda-forge
  six                                1.16.0        pyh6c4a22f_0             conda-forge
  stack_data                         0.6.2         pyhd8ed1ab_0             conda-forge
  statsmodels                        0.14.2        py311h18e1886_0          conda-forge
  stdlib-list                        0.10.0        pyhd8ed1ab_0             conda-forge
  sysroot_linux-64                   2.12          he073ed8_17              conda-forge
  tbb                                2021.12.0     h297d8ca_1               conda-forge
  threadpoolctl                      3.5.0         pyhc1e730c_0             conda-forge
  tk                                 8.6.13        noxft_h4845f30_101       conda-forge
  tktable                            2.10          h8bc8fbc_6               conda-forge
  toml                               0.10.2        pyhd8ed1ab_0             conda-forge
  tomlkit                            0.12.5        pyha770c72_0             conda-forge
  tornado                            6.4.1         py311h331c9d8_0          conda-forge
  tqdm                               4.66.4        pyhd8ed1ab_0             conda-forge
  traitlets                          5.14.3        pyhd8ed1ab_0             conda-forge
  typing_extensions                  4.12.2        pyha770c72_0             conda-forge
  tzdata                             2024a         h0c530f3_0               conda-forge
  tzlocal                            5.2           py311h38be061_0          conda-forge
  umap-learn                         0.5.5         py311h38be061_1          conda-forge
  wcwidth                            0.2.13        pyhd8ed1ab_0             conda-forge
  wheel                              0.43.0        pyhd8ed1ab_1             conda-forge
  xmltodict                          0.13.0        pyhd8ed1ab_0             conda-forge
  xz                                 5.2.6         h166bdaf_0               conda-forge
  yaml                               0.2.5         h7f98852_2               conda-forge
  yq                                 3.4.3         pyhd8ed1ab_0             conda-forge
  zeromq                             4.3.5         h75354e8_4               conda-forge
  zipp                               3.19.2        pyhd8ed1ab_0             conda-forge
  zlib                               1.3.1         h4ab18f5_1               conda-forge
  zstd                               1.5.6         ha6fb4c9_0               conda-forge
PauBadiaM commented 2 months ago

Hi @MarwaneBourdim,

You can transform your matrix to a csr_matrix for it to work properly:

from scipy.sparse import csr_matrix

adata.raw.X = csr_matrix(adata.raw.X)

Nonetheless, I think I fixed this isse in decoupler >= 1.6.0, which version do you have installed?

MarwaneBourdim commented 2 months ago

Hi @PauBadiaM,

Thank you for your prompt response. I've got the latest version of decoupler, when I try your code snippet I get this error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[18], line 1
----> 1 adata.raw.X = csr_matrix(adata.raw.X)

AttributeError: can't set attribute 'X'

Nonetheless, I dont think that's where the problem is because when I look at:

adata.raw.X

It outputs:

<Compressed Sparse Row sparse matrix of dtype 'float32'
    with 2238732 stored elements and shape (2638, 13714)>

However, if I try to look at its attribute A: adata.raw.X.A

I get:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[20], line 1
----> 1 adata.raw.X.A

AttributeError: 'csr_matrix' object has no attribute 'A'

Like in the original error.

Do you think, it's a problem with scipy? I even went to check the list of attributes of scipy sparse matrices, "A" wasn't explicitly in it but it was an attribute of "numpy.matrix" which is a datatype I don't get the above error with. I guess sparse matrices are defined from numpy matrices and that this attribute is supposed to be heritable for scipy sparse matrices but not in my environment for some reason? It seems really strange.

PauBadiaM commented 2 months ago

Could be that you get this error because you store your counts values as a sparse matrix and not a sparse array? These two classes, even though they look the same have different properties. In general, in the scverse ecosystem people use arrays over matrices.

MarwaneBourdim commented 2 months ago

I get the same error with sparse arrays:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[26], line 3
      1 import scipy as sp
----> 3 sp.sparse.csr_array(adata.raw.X).A
      4 sp.sparse.csr_matrix(adata.raw.X).A

AttributeError: 'csr_array' object has no attribute 'A'
PauBadiaM commented 2 months ago

Ahh oh wow, apparently in the newer scipy version they have deprecated the use of .A (see https://github.com/scipy/scipy/issues/21049#issuecomment-2189757313). I'll have to refactor some code but in the meantime you can downgrade scipy to < 1.14.0, thanks for reporting this!

MarwaneBourdim commented 2 months ago

Perfect thank you!