search

aertslab / pySCENIC

pySCENIC is a lightning-fast python implementation of the SCENIC pipeline (Single-Cell rEgulatory Network Inference and Clustering) which enables biologists to infer transcription factors, gene regulatory networks and cell types from single-cell RNA-seq data.
http://scenic.aertslab.org
GNU General Public License v3.0
439 stars 181 forks source link

Error running GRNBoost2 step #525

Open eblchen opened 9 months ago

eblchen commented 9 months ago

Error running: pyscenic grn \ --num_workers 20 \ --output adj.tsv \ --method grnboost2 \ PBMC10k_filtered.loom \ hs_hgnc_tfs.txt

(...)
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 269.80 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 263.60 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 267.21 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 268.83 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Heartbeat to scheduler failed
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 268.80 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Heartbeat to scheduler failed
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
(...)
Traceback (most recent call last):
  File "/home/echen/miniforge3/envs/scenic/bin/pyscenic", line 8, in <module>
    sys.exit(main())
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 675, in main
    args.func(args)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 100, in find_adjacencies_command
    expression_data=ex_mtx, tf_names=tf_names, verbose=True, client_or_address=client, seed=args.seed
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/arboreto/algo.py", line 41, in grnboost2
    early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/arboreto/algo.py", line 128, in diy
    seed=seed)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/arboreto/core.py", line 421, in create_graph
    [future_tf_matrix_gene_names] = client.scatter([tf_matrix_gene_names], broadcast=True)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/client.py", line 2175, in scatter
    hash=hash,
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/client.py", line 838, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils.py", line 351, in sync
    raise exc.with_traceback(tb)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils.py", line 334, in f
    result[0] = yield future
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/client.py", line 2062, in _scatter
    timeout=timeout,
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 660, in send_recv
    raise exc.with_traceback(tb)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 500, in handle_comm
    result = await result
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/scheduler.py", line 5043, in scatter
    await self.replicate(keys=keys, workers=workers, n=n)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/scheduler.py", line 5511, in replicate
    for w, who_has in gathers.items()
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils_comm.py", line 389, in retry_operation
    operation=operation,
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils_comm.py", line 369, in retry
    return await coro()
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 662, in send_recv
    raise Exception(response["text"])
Exception: 'data'

Package list:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aiohttp                   3.8.6                    pypi_0    pypi
aiosignal                 1.2.0                    pypi_0    pypi
alsa-lib                  1.2.7.2              h166bdaf_0    conda-forge
anndata                   0.7.8                    pypi_0    pypi
arboreto                  0.1.6                    pypi_0    pypi
arpack                    3.7.0                hdefa2d7_2    conda-forge
async-timeout             4.0.2                    pypi_0    pypi
asynctest                 0.13.0                   pypi_0    pypi
attrs                     22.2.0                   pypi_0    pypi
blosc                     1.21.5               h0f2a231_0    conda-forge
bokeh                     2.3.3                    pypi_0    pypi
boltons                   23.1.1                   pypi_0    pypi
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.25.0               hd590300_0    conda-forge
ca-certificates           2023.11.17           hbcca054_0    conda-forge
cached-property           1.5.2                    pypi_0    pypi
certifi                   2021.5.30        py36h5fab9bb_0    conda-forge
cffi                      1.14.6           py36hd8eec40_1    conda-forge
charset-normalizer        2.0.12                   pypi_0    pypi
click                     8.0.4                    pypi_0    pypi
cloudpickle               2.2.1                    pypi_0    pypi
contextvars               2.4                      pypi_0    pypi
ctxcore                   0.1.1                    pypi_0    pypi
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.11.0           py36h7b6447c_0    anaconda
dask                      2021.3.0                 pypi_0    pypi
dbus                      1.13.6               h5008d03_3    conda-forge
decorator                 4.4.2                    pypi_0    pypi
dill                      0.3.4                    pypi_0    pypi
distributed               2021.3.0                 pypi_0    pypi
expat                     2.5.0                hcb278e6_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
frozendict                2.4.0                    pypi_0    pypi
frozenlist                1.2.0                    pypi_0    pypi
fsspec                    2022.1.0                 pypi_0    pypi
get-version               2.1                      pypi_0    pypi
gettext                   0.21.1               h27087fc_0    conda-forge
glib                      2.78.3               hfc55251_0    conda-forge
glib-tools                2.78.3               hfc55251_0    conda-forge
glpk                      4.65              h9202a9a_1004    conda-forge
gmp                       6.3.0                h59595ed_0    conda-forge
gst-plugins-base          1.20.3               h57caac4_2    conda-forge
gstreamer                 1.20.3               hd4edc92_2    conda-forge
h5py                      3.1.0                    pypi_0    pypi
hdf5                      1.12.1          nompi_h2386368_104    conda-forge
heapdict                  1.0.1                    pypi_0    pypi
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.6                      pypi_0    pypi
idna-ssl                  1.1.0                    pypi_0    pypi
immutables                0.19                     pypi_0    pypi
importlib-metadata        4.8.3                    pypi_0    pypi
importlib-resources       5.4.0                    pypi_0    pypi
interlap                  0.2.7                    pypi_0    pypi
jinja2                    3.0.3                    pypi_0    pypi
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.3.1            py36h605e78d_1    conda-forge
krb5                      1.20.1               hf9c8cef_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
legacy-api-wrap           1.2                      pypi_0    pypi
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           20_linux64_openblas    conda-forge
libcblas                  3.9.0           20_linux64_openblas    conda-forge
libclang                  13.0.1          default_h7634d5b_5    conda-forge
libcurl                   7.87.0               h6312ad2_0    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_3    conda-forge
libgfortran5              13.2.0               ha4646dd_3    conda-forge
libglib                   2.78.3               h783c2da_0    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           20_linux64_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm13                 13.0.1               hf817b99_2    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.25          pthreads_h413a1c8_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     14.5                 h2baec63_5    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libtiff                   4.3.0                h0fcbabc_4    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.14               haae042b_4    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvmlite                  0.36.0           py36h05121d2_0    conda-forge
locket                    1.0.0                    pypi_0    pypi
loompy                    3.0.7                    pypi_0    pypi
louvain                   0.7.0            py36hf105032_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
markupsafe                2.0.1                    pypi_0    pypi
matplotlib                3.3.4            py36h5fab9bb_0    conda-forge
matplotlib-base           3.3.4            py36hd391965_0    conda-forge
metis                     5.1.0             h59595ed_1007    conda-forge
mock                      5.1.0              pyhd8ed1ab_0    conda-forge
mpfr                      4.2.1                h9458935_0    conda-forge
msgpack                   1.0.5                    pypi_0    pypi
multicore-tsne            0.1_d4ff4aab     py36h706143c_1    conda-forge
multidict                 5.2.0                    pypi_0    pypi
multiprocessing-on-dill   3.5.0a4                  pypi_0    pypi
mysql-common              8.0.32               h14678bc_0    conda-forge
mysql-libs                8.0.32               h54cf53e_0    conda-forge
natsort                   8.2.0                    pypi_0    pypi
ncurses                   6.4                  h59595ed_2    conda-forge
networkx                  2.5.1                    pypi_0    pypi
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.96                 h1d7d5a4_0    conda-forge
numba                     0.53.1           py36h1328434_1    conda-forge
numexpr                   2.7.3            py36h0cdc3f0_0    conda-forge
numpy                     1.19.5           py36hfc0c790_2    conda-forge
numpy-groupies            0.9.22                   pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.5.0                h7d73246_0    conda-forge
openssl                   1.1.1w               hd590300_0    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    1.1.5            py36h284efc9_0    conda-forge
partd                     1.2.0                    pypi_0    pypi
patsy                     0.5.6              pyhd8ed1ab_0    conda-forge
pcre2                     10.42                hcad00b1_0    conda-forge
pillow                    8.3.2            py36h676a545_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
psutil                    5.9.7                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyarrow                   0.16.0                   pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pynndescent               0.5.11                   pypi_0    pypi
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyqt                      5.12.3           py36h5fab9bb_7    conda-forge
pyqt-impl                 5.12.3           py36h7ec31b9_7    conda-forge
pyqt5-sip                 4.19.18          py36hc4f0c31_7    conda-forge
pyqtchart                 5.12             py36h7ec31b9_7    conda-forge
pyqtwebengine             5.12.1           py36h7ec31b9_7    conda-forge
pyscenic                  0.11.2                   pypi_0    pypi
pytables                  3.6.1            py36he0b5dfd_4    conda-forge
python                    3.6.15          hb7a2778_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-igraph             0.9.6            py36h644ed5e_0    conda-forge
python_abi                3.6                     2_cp36m    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1                    pypi_0    pypi
qt                        5.12.9               h1304e3e_6    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.27.1                   pypi_0    pypi
scanpy                    1.7.2                    pypi_0    pypi
scikit-learn              0.24.2           py36hc89565f_1    conda-forge
scipy                     1.5.3            py36h81d768a_1    conda-forge
seaborn                   0.11.2               hd8ed1ab_0    conda-forge
seaborn-base              0.11.2             pyhd8ed1ab_0    conda-forge
setuptools                58.0.4           py36h5fab9bb_2    conda-forge
sinfo                     0.3.4                    pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sortedcontainers          2.4.0                    pypi_0    pypi
sqlite                    3.44.2               h2c6b66d_0    conda-forge
statsmodels               0.12.2           py36he33b4a0_0    conda-forge
stdlib-list               0.8.0                    pypi_0    pypi
suitesparse               5.10.1               hd8046ac_0    conda-forge
tbb                       2020.2               h4bd325d_4    conda-forge
tblib                     1.7.0                    pypi_0    pypi
texttable                 1.7.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toolz                     0.11.2             pyhd3eb1b0_0    anaconda
tornado                   6.1              py36h8f6f2f9_1    conda-forge
tqdm                      4.64.1                   pypi_0    pypi
typing-extensions         4.1.1                    pypi_0    pypi
umap-learn                0.5.5                    pypi_0    pypi
urllib3                   1.26.18                  pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xlrd                      1.2.0                    pypi_0    pypi
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.7.2                    pypi_0    pypi
zict                      2.1.0                    pypi_0    pypi
zipp                      3.6.0                    pypi_0    pypi
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge
ghuls commented 9 months ago

Did you try with less number of workers?

eblchen commented 9 months ago

Tried with 10 workers, this was the end of the error message:

OSError: Timed out during handshake while connecting to tcp://127.0.0.1:45347 after 10 s
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 446.74 MB -- Worker memory limit: 637.97 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 446.74 MB -- Worker memory limit: 637.97 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 446.74 MB -- Worker memory limit: 637.97 MB
Killed
(scenic) echen@20766MOPLT:~/working/scenic$ /home/echen/miniforge3/envs/scenic/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 48 leaked semaphores to clean up at shutdown
  len(cache))
ghuls commented 9 months ago

How much RAM do your have? You might need to lower the number of threads even more. Also note that we only run pySCENIC on servers and not on local computers (which might not have enough RAM).

eblchen commented 9 months ago

Ah I see I'll try on a server then, thank you

Ocean-Lyu commented 7 months ago

Hi, I also encountered a problem when running GRNBoost2 step:

pyscenic grn --num_workers 20 \ --seed 42 \ --method grnboost2 \ --output ./results/grn_def.csv \ sndef.csv allTFs_mm.txt

2024-03-22 10:36:57,207 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2024-03-22 10:38:27,167 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
Traceback (most recent call last):
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/bin/pyscenic", line 8, in <module>
    sys.exit(main())
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
    args.func(args)
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 106, in find_adjacencies_command
    network = method(
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/arboreto/algo.py", line 39, in grnboost2
    return diy(expression_data=expression_data, regressor_type='GBM', regressor_kwargs=SGBM_KWARGS,
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/arboreto/algo.py", line 120, in diy
    graph = create_graph(expression_matrix,
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/arboreto/core.py", line 450, in create_graph
    all_meta_df = from_delayed(delayed_meta_dfs, meta=_META_SCHEMA)
  File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/dask_expr/io/_delayed.py", line 93, in from_delayed
    raise TypeError("Must supply at least one delayed object")
TypeError: Must supply at least one delayed object

I am using pySCENIC 0.12.1+2.geaf23eb on a 24 core 1000G RAM HPC

package list 
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
anndata                   0.10.6                   pypi_0    pypi
arboreto                  0.1.6                    pypi_0    pypi
array-api-compat          1.5.1                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
bokeh                     3.4.0                    pypi_0    pypi
boltons                   23.1.1                   pypi_0    pypi
bzip2                     1.0.8                h5eee18b_5    defaults
ca-certificates           2024.3.11            h06a4308_0    defaults
certifi                   2024.2.2                 pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
comm                      0.2.2                    pypi_0    pypi
contourpy                 1.2.0                    pypi_0    pypi
ctxcore                   0.2.0                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cytoolz                   0.12.3                   pypi_0    pypi
dask                      2024.3.1                 pypi_0    pypi
dask-expr                 1.0.4                    pypi_0    pypi
debugpy                   1.8.1                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
distributed               2024.3.1                 pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
fonttools                 4.50.0                   pypi_0    pypi
frozendict                2.4.0                    pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.3.1                 pypi_0    pypi
h5py                      3.10.0                   pypi_0    pypi
idna                      3.6                      pypi_0    pypi
importlib-metadata        7.1.0                    pypi_0    pypi
interlap                  0.2.7                    pypi_0    pypi
ipykernel                 6.29.3                   pypi_0    pypi
ipython                   8.22.2                   pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.3.2                    pypi_0    pypi
jupyter-client            8.6.1                    pypi_0    pypi
jupyter-core              5.7.2                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1    defaults
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
llvmlite                  0.42.0                   pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
loompy                    3.0.7                    pypi_0    pypi
lz4                       4.3.3                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.8.3                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
msgpack                   1.0.8                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multiprocessing-on-dill   3.5.0a4                  pypi_0    pypi
natsort                   8.4.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0    defaults
nest-asyncio              1.6.0                    pypi_0    pypi
networkx                  3.2.1                    pypi_0    pypi
numba                     0.59.1                   pypi_0    pypi
numexpr                   2.9.0                    pypi_0    pypi
numpy                     1.26.4                   pypi_0    pypi
numpy-groupies            0.10.2                   pypi_0    pypi
openssl                   3.0.13               h7f8727e_0    defaults
packaging                 24.0                     pypi_0    pypi
pandas                    2.2.1                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
partd                     1.4.1                    pypi_0    pypi
patsy                     0.5.6                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       23.3.1                   pypi_0    pypi
platformdirs              4.2.0                    pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyarrow                   15.0.2                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pynndescent               0.5.11                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pyscenic                  0.12.1+2.geaf23eb          pypi_0    pypi
python                    3.10.13              h955ad1f_0    defaults
python-dateutil           2.9.0.post0              pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
pyzmq                     25.1.2                   pypi_0    pypi
readline                  8.2                  h5eee18b_0    defaults
requests                  2.31.0                   pypi_0    pypi
scanpy                    1.9.8                    pypi_0    pypi
scikit-learn              1.4.1.post1              pypi_0    pypi
scipy                     1.12.0                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
session-info              1.0.0                    pypi_0    pypi
setuptools                68.2.2                   pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0    defaults
stack-data                0.6.3                    pypi_0    pypi
statsmodels               0.14.1                   pypi_0    pypi
stdlib-list               0.10.0                   pypi_0    pypi
tblib                     3.0.0                    pypi_0    pypi
threadpoolctl             3.4.0                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0    defaults
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4                      pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
traitlets                 5.14.2                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
umap-learn                0.5.5                    pypi_0    pypi
urllib3                   2.2.1                    pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
wheel                     0.41.2                   pypi_0    pypi
xyzservices               2023.10.1                pypi_0    pypi
xz                        5.4.6                h5eee18b_0    defaults
yarl                      1.9.4                    pypi_0    pypi
zict                      3.0.0                    pypi_0    pypi
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0    defaults
PaulineLX commented 7 months ago

I met the same problem. I guessed it was probably due to version incompatibility. So I changed the version of packages. However, it didn't work. And when I input as follows, it ran out results. pyscenic ctx --num_workers 10 \ --output regulons.csv \ --expression_mtx_fname sce.loom \ --mode "custom_multiprocessing" \ --annotations_fname motifs-v9-nr.hgnc-m0.001-o0.0.tbl \ sce.adj.csv \ hg38_10kbp_up_10kbp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather

malihhhh commented 7 months ago

Ocean-Lyu I met the same problem as you. But I did't find a resolution.

peipp410 commented 7 months ago

Ocean-Lyu I met the same problem as you. But I did't find a resolution.

Also the same problem. It seems that this issue has only arisen recently.

r-trimbour commented 7 months ago

Changing dask/distributed version worked for me. You can try pip install distributed==2023.12.1 !

dactylogram commented 7 months ago

If you got error message of "TypeError: Must supply at least one delayed object", It may be caused by dependency error of dask-expr package. dask-expr package was currently updated from version 0.5.3 (Feb 28) to 1.0 (Mar 12)

You could avoid this message by down-grading dask-expr package, by following code pip install dask-expr==0.5.3

ghuls commented 7 months ago

git version of arboreto should fix this: https://github.com/aertslab/arboreto/pull/39

IMT-Atlantic commented 7 months ago

If you got error message of "TypeError: Must supply at least one delayed object", It may be caused by dependency error of dask-expr package. dask-expr package was currently updated from version 0.5.3 (Feb 28) to 1.0 (Mar 12)

You could avoid this message by down-grading dask-expr package, by following code pip install dask-expr==0.5.3

Merci bcp!!! C'etait Vrai!!!

yifanfu01 commented 6 months ago

Hi guys, I used HPC without Root access. and I use conda environment and command line of pyscenic. I solve this problems by @r-trimbour and @dactylogram their answer: pip install dask-expr==0.5.3 distributed==2024.2.1 only install on package will cause "ERROR: pip's dependency resolver does not currently take into account all..."

IMT-Atlantic commented 6 months ago

Hi guys, I used HPC without Root access. and I use conda environment and command line of pyscenic. I solve this problems by @r-trimbour and @dactylogram their answer: pip install dask-expr==0.5.3 distributed==2024.2.1 only install on package will cause "ERROR: pip's dependency resolver does not currently take into account all..." oui, mais je sais pas pourquoi a cette question

r-trimbour commented 6 months ago

Hi @yifanfu01 I don't know if you're still struggling with the installation, but if it's the case I think dask depends on distributed so you have to choose the same version for both of them. We were both suggesting to update their versions but you can choose as long as it's before marche 2024 on pip from what I remember. :)

liezeltamon commented 5 months ago

git version of arboreto should fix this: aertslab/arboreto#39

running into this issue, https://github.com/aertslab/arboreto/issues/33, when trying to install arboreto from source..

ASNbioinf commented 4 months ago

I have a similar problem plus "ERROR - Error in Nanny killing Worker subprocess". I can't understand if it's because I put "--rm" as the argument. It automatically removes the container once it exists. Moreover, the problem was apparently resolved below by the explanation "The above exception was the direct cause of the following exception". Help me understand the error so I can figure out how I need to run the analysis again.

I use 150GB and 20 CPU 

docker run -it --rm \
    -v /data:/data \
    aertslab/pyscenic:0.12.1 pyscenic grn \
        --num_workers 20 \
        -o /data/expr_mat.adjacencies.tsv \
        /data/expr_mat.loom \
        /data/allTFs_hg38.txt

024-06-25 21:09:47,005 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks. 2024-06-26 16:12:16,777 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,778 - distributed.nanny - WARNING - Worker process still alive after 3.1999989318847657 seconds, killing 2024-06-26 16:12:16,778 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,778 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.1999986267089846 seconds, killing 2024-06-26 16:12:16,780 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2024-06-26 16:12:16,780 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,780 - distributed.nanny - WARNING - Worker process still alive after 3.1999989318847657 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.199998474121094 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:17,493 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,495 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,496 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,498 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,499 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,503 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,542 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,545 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,568 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,570 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError

2024-06-26 16:12:17,576 - pyscenic.cli.pyscenic - INFO - Writing results to file. preparing dask client parsing input creating dask graph 20 partitions computing dask graph not shutting down client, client was created externally finished

CC-DrDU commented 2 months ago

Hi guys, I used HPC without Root access. and I use conda environment and command line of pyscenic. I solve this problems by @r-trimbour and @dactylogram their answer: pip install dask-expr==0.5.3 distributed==2024.2.1 only install on package will cause "ERROR: pip's dependency resolver does not currently take into account all..."

Thanks a lot! I've solved the problem. But I also noticed that this solution did not work out under python 3.11.9. If anybody fail to solve this problem, you could try again under python 3.11.8 which worked just fine for me.