arboreto_with_multiprocessing.py fails for every target

[noranekonobokkusu]

Hi,

arboreto_with_multiprocessing.py doesn't see any targets, which I was able to reproduce on a test 10x dataset.

Steps to reproduce the behavior

1. In Jupyter notebook, I created 10x_test.loom file on a test dataset following this tutorial

2. Running pyscenic from within Jupyter notebook crashes for me (as in this issue), so I tried running the command line:

arboreto_with_multiprocessing.py --num_worker 4 --sparse --method grnboost2 -o 10x_test.adj.csv 10x_test.loom /path/to/allTFs_hg38.txt

The output produces warning for every single gene in the object, starting as

Loaded expression matrix of 11769 cells and 33538 genes in 7.553167104721069 seconds...
Loaded 1892 TFs...
starting grnboost2 using 4 processes...
  0%|                                                                                                                             | 0/33538 [00:00<?, ?it/s]'WARNING: infer_data failed for target MIR1302-2HG' Retry (1/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (2/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (3/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (4/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (5/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (6/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (7/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (8/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (9/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target MIR1302-2HG' Retry (10/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
  0%|                                                                                                                  | 1/33538 [00:03<34:40:56,  3.72s/it]'WARNING: infer_data failed for target FAM138A' Retry (1/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (2/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (3/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (4/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (5/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (6/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (7/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (8/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (9/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
'WARNING: infer_data failed for target FAM138A' Retry (10/10). Failure caused by AttributeError("'csc_matrix' object has no attribute 'A'").
  0%|                                                                                                                  | 2/33538 [00:04<16:41:08,  1.79s/it]

Expected behavior I expected a non-empty output of this command. My output is a single line in 10x_test.adj.csv: TF,target,importance

Please complete the following information:

• pySCENIC version: 0.12.1+2.geaf23eb
• Installation method: pip in virtual environment
• Run environment: shell terminal
• OS: macOS Sonoma 14.5 (23F79)
• Package versions:

  aiohappyeyeballs==2.3.4
  aiohttp==3.10.0
  aiosignal==1.3.1
  anndata==0.10.8
  anyio==4.4.0
  appnope==0.1.4
  arboreto==0.1.6
  argon2-cffi==23.1.0
  argon2-cffi-bindings==21.2.0
  array_api_compat==1.8
  arrow==1.3.0
  asttokens==2.4.1
  async-lru==2.0.4
  attrs==23.2.0
  Babel==2.15.0
  beautifulsoup4==4.12.3
  bleach==6.1.0
  bokeh==3.5.1
  boltons==24.0.0
  certifi==2024.7.4
  cffi==1.16.0
  charset-normalizer==3.3.2
  click==8.1.7
  cloudpickle==3.0.0
  comm==0.2.2
  contourpy==1.2.1
  ctxcore==0.2.0
  cycler==0.12.1
  cytoolz==0.12.3
  dask==2024.7.1
  dask-expr==1.1.9
  debugpy==1.8.2
  decorator==5.1.1
  defusedxml==0.7.1
  dill==0.3.8
  distributed==2024.7.1
  executing==2.0.1
  fastjsonschema==2.20.0
  fonttools==4.53.1
  fqdn==1.5.1
  frozendict==2.4.4
  frozenlist==1.4.1
  fsspec==2024.6.1
  h11==0.14.0
  h5py==3.11.0
  httpcore==1.0.5
  httpx==0.27.0
  idna==3.7
  importlib_metadata==8.2.0
  interlap==0.2.7
  ipykernel==6.29.5
  ipython==8.26.0
  isoduration==20.11.0
  jedi==0.19.1
  Jinja2==3.1.4
  joblib==1.4.2
  json5==0.9.25
  jsonpointer==3.0.0
  jsonschema==4.23.0
  jsonschema-specifications==2023.12.1
  jupyter-events==0.10.0
  jupyter-lsp==2.2.5
  jupyter_client==8.6.2
  jupyter_core==5.7.2
  jupyter_server==2.14.2
  jupyter_server_terminals==0.5.3
  jupyterlab==4.2.4
  jupyterlab_pygments==0.3.0
  jupyterlab_server==2.27.3
  kiwisolver==1.4.5
  legacy-api-wrap==1.4
  llvmlite==0.43.0
  locket==1.0.0
  loompy==3.0.7
  lz4==4.3.3
  MarkupSafe==2.1.5
  matplotlib==3.9.1
  matplotlib-inline==0.1.7
  mistune==3.0.2
  msgpack==1.0.8
  multidict==6.0.5
  multiprocessing_on_dill==3.5.0a4
  natsort==8.4.0
  nbclient==0.10.0
  nbconvert==7.16.4
  nbformat==5.10.4
  nest-asyncio==1.6.0
  networkx==3.3
  notebook==7.2.1
  notebook_shim==0.2.4
  numba==0.60.0
  numexpr==2.10.1
  numpy==1.26.4
  numpy-groupies==0.11.2
  overrides==7.7.0
  packaging==24.1
  pandas==2.2.2
  pandocfilters==1.5.1
  parso==0.8.4
  partd==1.4.2
  patsy==0.5.6
  pexpect==4.9.0
  pillow==10.4.0
  platformdirs==4.2.2
  prometheus_client==0.20.0
  prompt_toolkit==3.0.47
  psutil==6.0.0
  ptyprocess==0.7.0
  pure_eval==0.2.3
  pyarrow==17.0.0
  pyarrow-hotfix==0.6
  pycparser==2.22
  Pygments==2.18.0
  pynndescent==0.5.13
  pyparsing==3.1.2
  pyscenic @ git+https://github.com/aertslab/pySCENIC@eaf23eb1fdcaae79b273de56b374b71aa8afde5a
  python-dateutil==2.9.0.post0
  python-json-logger==2.0.7
  pytz==2024.1
  PyYAML==6.0.1
  pyzmq==26.0.3
  referencing==0.35.1
  requests==2.32.3
  rfc3339-validator==0.1.4
  rfc3986-validator==0.1.1
  rpds-py==0.19.1
  scanpy==1.10.2
  scikit-learn==1.5.1
  scipy==1.14.0
  seaborn==0.13.2
  Send2Trash==1.8.3
  session_info==1.0.0
  setuptools==72.1.0
  six==1.16.0
  sniffio==1.3.1
  sortedcontainers==2.4.0
  soupsieve==2.5
  stack-data==0.6.3
  statsmodels==0.14.2
  stdlib-list==0.10.0
  tblib==3.0.0
  terminado==0.18.1
  threadpoolctl==3.5.0
  tinycss2==1.3.0
  toolz==0.12.1
  tornado==6.4.1
  tqdm==4.66.4
  traitlets==5.14.3
  types-python-dateutil==2.9.0.20240316
  tzdata==2024.1
  umap-learn==0.5.6
  uri-template==1.3.0
  urllib3==2.2.2
  wcwidth==0.2.13
  webcolors==24.6.0
  webencodings==0.5.1
  websocket-client==1.8.0
  xyzservices==2024.6.0
  yarl==1.9.4
  zict==3.0.0
  zipp==3.19.2

[ghuls]

Install an earlier version of scipy than 1.14.0:

Several previously deprecated methods for sparse arrays were removed: asfptype, getrow, getcol, get_shape, getmaxprint, set_shape, getnnz, and getformat. Additionally, the .A and .H attributes were removed.

https://docs.scipy.org/doc/scipy/release/1.14.0-notes.html