search

aertslab / pySCENIC

pySCENIC is a lightning-fast python implementation of the SCENIC pipeline (Single-Cell rEgulatory Network Inference and Clustering) which enables biologists to infer transcription factors, gene regulatory networks and cell types from single-cell RNA-seq data.
http://scenic.aertslab.org
GNU General Public License v3.0
438 stars 181 forks source link

pyscenic ctx gives a TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given and an EOFError #425

Closed JewelBanik closed 2 years ago

JewelBanik commented 2 years ago

Hi,

I am trying to analyze my scRNA-seq data using pySCENIC (v.0.12.0) by following this guideline http://htmlpreview.github.io/?https://github.com/aertslab/SCENICprotocol/blob/master/notebooks/PBMC10k_SCENIC-protocol-CLI.html.

At the SCENIC analysis step, I am stuck at the regulon prediction step which gives me a typeError and and EOFError.

I have been trying for a few days to get around this but I couldn't. Any help is highly appreciated. Please note that I am using feather v2 databases, and my adj.tsv file looks OK.

Many thanks for your help!

The code that gives me the error: 

f_db_names
'C:\\Users\\3054501\\Desktop\\odle_mutant\\databases\\mm10_10kbp_up_10kbp_down_full_tx_clustered.genes_vs_motifs.rankings.feather C:\\Users\\3054501\\Desktop\\odle_mutant\\databases\\mm10_500bp_up_100bp_down_full_tx_clustered.genes_vs_motifs.rankings.feather'

f_motif_path
'C:\\Users\\3054501\\Desktop\\odle_mutant\\motifs-v9-nr.mgi-m0.001-o0.0.tbl'

!pyscenic ctx adj.tsv \
    {f_db_names} \
    --annotations_fname {f_motif_path} \ 
    --expression_mtx_fname {'scrna_filt.loom'} \
    --output reg.csv \
    --mask_dropouts \
    --num_workers 8
...
  1. Error encountered:

2022-09-16 08:17:47,708 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2022-09-16 08:17:49,869 - pyscenic.utils - INFO - Calculating Pearson correlations.

2022-09-16 08:17:50,383 - pyscenic.utils - WARNING - Note on correlation calculation: the default behaviour for calculating the correlations has changed after pySCENIC verion 0.9.16. Previously, the default was to calculate the correlation between a TF and target gene using only cells with non-zero expression values (mask_dropouts=True). The current default is now to use all cells to match the behavior of the R verision of SCENIC. The original settings can be retained by setting 'rho_mask_dropouts=True' in the modules_from_adjacencies function, or '--mask_dropouts' from the CLI. Dropout masking is currently set to [True].

2022-09-16 08:18:17,527 - pyscenic.utils - INFO - Creating modules.

2022-09-16 08:19:16,685 - pyscenic.cli.pyscenic - INFO - Loading databases.

2022-09-16 08:19:16,839 - pyscenic.cli.pyscenic - INFO - Calculating regulons.

2022-09-16 08:19:16,839 - pyscenic.prune - INFO - Using 8 workers.

2022-09-16 08:19:16,839 - pyscenic.prune - INFO - Using 8 workers. Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "", line 1, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\multiprocessing\spawn.py", line 129, in _main return self._bootstrap(parent_sentinel) TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given Traceback (most recent call last): File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\multiprocessing_on_dill\connection.py", line 301, in _recv_bytes ov, err = _winapi.ReadFile(self._handle, bsize, BrokenPipeError: [WinError 109] The pipe has been ended

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\3054501\Anaconda3\envs\Odle\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\3054501\Anaconda3\envs\Odle\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\3054501\Anaconda3\envs\Odle\Scripts\pyscenic.exe__main__.py", line 7, in File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\pyscenic\cli\pyscenic.py", line 677, in main args.func(args) File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\pyscenic\cli\pyscenic.py", line 221, in prune_targets_command df_motifs = calc_func( File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\pyscenic\prune.py", line 400, in prune2df return _distributed_calc( File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\pyscenic\prune.py", line 233, in _distributed_calc fnames = [recv.recv() for recv in receivers] File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\pyscenic\prune.py", line 233, in fnames = [recv.recv() for recv in receivers] File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\multiprocessing_on_dill\connection.py", line 250, in recv buf = self._recv_bytes() File "C:\Users\3054501\Anaconda3\envs\Odle\lib\site-packages\multiprocessing_on_dill\connection.py", line 321, in _recv_bytes raise EOFError EOFError ​ ...


**Please complete the following information:**
- pySCENIC version: 0.12.0
- Installation method: Pip
- Run environment: Jupyter notebook via Conda
- OS: Windows 10, version: 10.0.19041 Build 19041, x64 based PC, RAM 128GB, 11th Gen Intel(R) core(TM) i9-11900@2.5-GHz, 2496Mhz, 8core(s), 16logical processors
- Package versions:

packages in environment at C:\Users\3054501\Anaconda3\envs\Odle:

#

Name Version Build Channel

aiohttp 3.8.1 pypi_0 pypi aiosignal 1.2.0 pypi_0 pypi anndata 0.8.0 pypi_0 pypi anyio 3.5.0 py310haa95532_0
arboreto 0.1.6 pypi_0 pypi argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py310h2bbff1b_0
asttokens 2.0.5 pyhd3eb1b0_0
async-timeout 4.0.2 pypi_0 pypi attrs 22.1.0 pypi_0 pypi babel 2.9.1 pyhd3eb1b0_0
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.11.1 py310haa95532_0
bleach 4.1.0 pyhd3eb1b0_0
bokeh 2.4.3 pypi_0 pypi boltons 21.0.0 pypi_0 pypi brotlipy 0.7.0 py310h2bbff1b_1002
bzip2 1.0.8 he774522_0
ca-certificates 2022.9.14 h5b45459_0 conda-forge certifi 2022.9.14 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h2bbff1b_0
charset-normalizer 2.1.1 pypi_0 pypi click 8.1.3 pypi_0 pypi cloudpickle 2.2.0 pypi_0 pypi colorama 0.4.5 py310haa95532_0
cryptography 37.0.1 py310h21b164f_0
ctxcore 0.2.0 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cytoolz 0.12.0 pypi_0 pypi dask 2022.9.0 pypi_0 pypi debugpy 1.5.1 py310hd77b12b_0
decorator 5.1.1 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
dill 0.3.5.1 pypi_0 pypi distributed 2022.9.0 pypi_0 pypi entrypoints 0.4 py310haa95532_0
executing 0.8.3 pyhd3eb1b0_0
fonttools 4.37.1 pypi_0 pypi frozendict 2.3.4 pypi_0 pypi frozenlist 1.3.1 pypi_0 pypi fsspec 2022.8.2 pypi_0 pypi glib 2.69.1 h5dc1a3c_1
gst-plugins-base 1.18.5 h9e645db_0
gstreamer 1.18.5 hd78058f_0
h5py 3.7.0 pypi_0 pypi heapdict 1.0.1 pypi_0 pypi icu 58.2 ha925a31_3
idna 3.4 pypi_0 pypi igraph 0.9.11 pypi_0 pypi intel-openmp 2022.1.0 h57928b3_3787 conda-forge interlap 0.2.7 pypi_0 pypi ipykernel 6.15.2 py310haa95532_0
ipython 8.4.0 py310haa95532_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
ipywidgets 7.6.5 pyhd3eb1b0_1
jedi 0.18.1 py310haa95532_1
jinja2 3.1.2 pypi_0 pypi joblib 1.1.0 pypi_0 pypi jpeg 9e h2bbff1b_0
json5 0.9.6 pyhd3eb1b0_0
jsonschema 4.4.0 py310haa95532_0
jupyter 1.0.0 py310haa95532_8
jupyter_client 7.3.5 py310haa95532_0
jupyter_console 6.4.3 pyhd3eb1b0_0
jupyter_core 4.10.0 py310haa95532_0
jupyter_server 1.18.1 py310haa95532_0
jupyterlab 3.4.4 py310haa95532_0
jupyterlab_pygments 0.1.2 py_0
jupyterlab_server 2.12.0 py310haa95532_0
jupyterlab_widgets 1.0.0 pyhd3eb1b0_1
kiwisolver 1.4.4 pypi_0 pypi leidenalg 0.8.10 pypi_0 pypi libblas 3.9.0 16_win64_mkl conda-forge libcblas 3.9.0 16_win64_mkl conda-forge libclang 12.0.0 default_h627e005_2
libffi 3.4.2 hd77b12b_4
libiconv 1.16 h2bbff1b_2
liblapack 3.9.0 16_win64_mkl conda-forge libogg 1.3.5 h2bbff1b_1
libpng 1.6.37 h2a8f88b_0
libsodium 1.0.18 h62dcd97_0
libvorbis 1.3.7 he774522_0
libwebp 1.2.2 h2bbff1b_0
libxml2 2.9.14 h0ad7f3c_0
libxslt 1.1.35 h2bbff1b_0
llvmlite 0.39.1 pypi_0 pypi locket 1.0.0 pypi_0 pypi loompy 3.0.7 pypi_0 pypi louvain 0.7.1 pypi_0 pypi lz4-c 1.9.3 h2bbff1b_1
markupsafe 2.1.1 py310h2bbff1b_0
matplotlib 3.5.3 pypi_0 pypi matplotlib-inline 0.1.6 py310haa95532_0
mistune 0.8.4 py310h2bbff1b_1000
mkl 2022.1.0 h6a75c08_874 conda-forge msgpack 1.0.4 pypi_0 pypi multicore-tsne 0.1_d4ff4aab py310hc781a3c_2 conda-forge multidict 6.0.2 pypi_0 pypi multiprocess 0.70.13 pypi_0 pypi multiprocessing-on-dill 3.5.0a4 pypi_0 pypi natsort 8.2.0 pypi_0 pypi nbclassic 0.3.5 pyhd3eb1b0_0
nbclient 0.5.13 py310haa95532_0
nbconvert 6.4.4 py310haa95532_0
nbformat 5.3.0 py310haa95532_0
nest-asyncio 1.5.5 py310haa95532_0
networkx 2.8.6 pypi_0 pypi notebook 6.4.12 py310haa95532_0
numba 0.56.2 pypi_0 pypi numexpr 2.8.3 pypi_0 pypi numpy 1.23.3 pypi_0 pypi numpy-groupies 0.9.19 pypi_0 pypi openssl 1.1.1q h8ffe710_0 conda-forge packaging 21.3 pyhd3eb1b0_0
pandas 1.4.4 pypi_0 pypi pandocfilters 1.5.0 pyhd3eb1b0_0
parso 0.8.3 pyhd3eb1b0_0
partd 1.3.0 pypi_0 pypi patsy 0.5.2 pypi_0 pypi pcre 8.45 hd77b12b_0
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.2.0 pypi_0 pypi pip 22.1.2 py310haa95532_0
ply 3.11 py310haa95532_0
prometheus_client 0.14.1 py310haa95532_0
prompt-toolkit 3.0.20 pyhd3eb1b0_0
prompt_toolkit 3.0.20 hd3eb1b0_0
psutil 5.9.2 pypi_0 pypi pure_eval 0.2.2 pyhd3eb1b0_0
pyarrow 9.0.0 pypi_0 pypi pycparser 2.21 pyhd3eb1b0_0
pygments 2.11.2 pyhd3eb1b0_0
pynndescent 0.5.7 pypi_0 pypi pyopenssl 22.0.0 pyhd3eb1b0_0
pyparsing 3.0.9 py310haa95532_0
pyqt 5.15.7 py310hd77b12b_0
pyqt5-sip 12.11.0 py310hd77b12b_0
pyrsistent 0.18.0 py310h2bbff1b_0
pyscenic 0.12.0 pypi_0 pypi pysocks 1.7.1 py310haa95532_0
python 3.10.4 hbb2ffb3_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python-fastjsonschema 2.16.2 py310haa95532_0
python_abi 3.10 2_cp310 conda-forge pytz 2022.2.1 pypi_0 pypi pywin32 302 py310h2bbff1b_2
pywinpty 2.0.2 py310h5da7b33_0
pyyaml 6.0 pypi_0 pypi pyzmq 23.2.0 py310hd77b12b_0
qt-main 5.15.2 he8e5bd7_7
qt-webengine 5.15.9 hb9a9bb5_4
qtconsole 5.3.2 py310haa95532_0
qtpy 2.2.0 py310haa95532_0
qtwebkit 5.212 h3ad3cdb_4
requests 2.28.1 py310haa95532_0
scanpy 1.9.1 pypi_0 pypi scikit-learn 1.1.2 pypi_0 pypi scipy 1.9.1 pypi_0 pypi seaborn 0.12.0 pypi_0 pypi send2trash 1.8.0 pyhd3eb1b0_1
session-info 1.0.0 pypi_0 pypi setuptools 59.8.0 pypi_0 pypi sip 6.6.2 py310hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
sniffio 1.2.0 py310haa95532_1
sortedcontainers 2.4.0 pypi_0 pypi soupsieve 2.3.1 pyhd3eb1b0_0
sqlite 3.39.2 h2bbff1b_0
stack_data 0.2.0 pyhd3eb1b0_0
statsmodels 0.13.2 pypi_0 pypi stdlib-list 0.8.0 pypi_0 pypi tbb 2021.5.0 h2d74725_1 conda-forge tblib 1.7.0 pypi_0 pypi terminado 0.13.1 py310haa95532_0
testpath 0.6.0 py310haa95532_0
texttable 1.6.4 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tk 8.6.12 h2bbff1b_0
toml 0.10.2 pyhd3eb1b0_0
toolz 0.12.0 pypi_0 pypi tornado 6.1 pypi_0 pypi tqdm 4.64.1 pypi_0 pypi traitlets 5.1.1 pyhd3eb1b0_0
typing-extensions 4.3.0 py310haa95532_0
typing_extensions 4.3.0 py310haa95532_0
tzdata 2022c h04d1e81_0
umap-learn 0.5.3 pypi_0 pypi urllib3 1.26.12 pypi_0 pypi vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wcwidth 0.2.5 pyhd3eb1b0_0
webencodings 0.5.1 py310haa95532_1
websocket-client 0.58.0 py310haa95532_4
wheel 0.37.1 pyhd3eb1b0_0
widgetsnbextension 3.5.2 py310haa95532_0
win_inet_pton 1.1.0 py310haa95532_0
wincertstore 0.2 py310haa95532_2
winpty 0.4.3 4
xz 5.2.5 h8cc25b3_1
yarl 1.8.1 pypi_0 pypi zeromq 4.3.4 hd77b12b_0
zict 2.2.0 pypi_0 pypi zlib 1.2.12 h8cc25b3_3
zstd 1.5.2 h19a0ad4_0

...

ghuls commented 2 years ago

pySCENIC is never tested on Windows, but only on Linux as our compute infrastructure is all Linux (and bioinformatics in general is quite Linux based).

JewelBanik commented 2 years ago

Hi @ghuls, thanks so much for the clarification. I analyzed my other samples using SCENIC in R on Windows, but it was excruciatingly slow. However, I didn't know that pySCENIC is not Windows-compatible. Now, it makes sense to me. I will try on Linux, and let's see how it goes. Thanks again!

hyjforesight commented 2 years ago

Hello @ghuls, pyscenic ctx can run in Windows in Jupyter notebook when pyscenic version is v0.11.2. This BUG was caused by the update of pySCENIC. v0.11.2 works fine. v0.20 has this Bug. Could you please fix it? Thanks! Best, YJ

kwahcarioca commented 1 year ago

I couldnt solve the problem but it seems to be a conflict between multiprocessing_on_dill and multiprocessing packages. spawn.py of the first calls _Booststrap with one parameter parameter_sentinel but spawn.py of multiprocessing_on_dill calls _Booststrap with no parameter. I ve had a similar problem in other application running the package optix. If self is considered as one parameter so he could expect one or two parameters in accordance with the specific spawn.py version.

razorofockham commented 1 year ago

@JewelBanik @hyjforesight @kwahcarioca: I was getting the exact same error running pySCENIC v0.12.1 via CLI commands on a Windows conda environment, and managed to circumvent it by explicitly adding the flag --mode dask_multiprocessing in the call to pyscenic ctx. If you are still experiencing this issue, give that a shot to see if it solves it. In case it helps, my pySCENIC v0.12.1 conda environment is running python v3.10.12 with dask v2023.9.1 and multiprocessing-on-dill v3.5.0a4. Cheers, -Juan