Pyscenic ctx taking a long time

[willrosenow]

I have been running pyscenic ctx for over 5 days. My unfiltered loom file contains ~7k cells. How long does this step typically take? I am running it from a slurm script on my HPC using the following script:

pyscenic ctx files/adj.csv \ cisTarget_databases/mm10refseq-r80500bp_up_and_100bp_down_tss.mc9nr.feather \ --annotations_fname cisTarget_databases/motifs-v9-nr.mgi-m0.001-o0.0.tbl.txt \ --expression_mtx_fname files/ob_unfiltered.loom \ --output files/ob_pyscenic.csv \ --mask_dropouts \ --num_workers 39

In the output, it gets to the Calculating regulons step, but stops there.

2021-02-08 08:30:12,939 - pyscenic.cli.pyscenic - INFO - Calculating regulons.

Do you have any recommendations on making this script run faster?

Thanks, Will

[cflerin]

There's not really enough information here to help. At minimum you should provide the following from the bug report template:

• pySCENIC version: [e.g. 0.10.0, use pyscenic -h at the command line]
• Installation method: [e.g. Conda, Pip, Docker, Singularity]
• Run environment: [e.g. Jupyter notebook, CLI, Nextflow, HPC script]
• OS: [e.g. Ubuntu]
• Package versions: [obtain using pip freeze, conda list, or skip this if using Docker/Singularity]:

  ...

[willrosenow]

Here are the details:

pySCENIC version: 0.10.4

Installation method: Pip

Run environment: HPC slurm script

OS: CentOS

Package versions: adjustText==0.7.3 alabaster==0.7.12 anaconda-client==1.7.2 anaconda-navigator==1.9.7 anaconda-project==0.8.3 anndata==0.7.5 arboreto==0.1.5 asn1crypto==1.0.1 astroid==2.3.1 astropy==3.2.2 atomicwrites==1.3.0 attrs==19.2.0 Babel==2.7.0 backcall==0.1.0 backports.os==0.1.1 backports.shutil-get-terminal-size==1.0.0 beautifulsoup4==4.8.0 biothings-client==0.2.3 bitarray==1.0.1 bkcharts==0.2 blaze==0.11.3 bleach==3.1.0 bokeh==1.3.4 boltons==20.1.0 boto==2.49.0 Bottleneck==1.2.1 cellbrowser==0.7.15 cellex==1.2.1 certifi==2019.9.11 cffi==1.12.3 chardet==3.0.4 Click==7.0 cloudpickle==1.2.2 clyent==1.2.2 colorama==0.4.1 conda==4.8.0 conda-build==3.18.11 conda-package-handling==1.6.0 conda-verify==3.1.1 contextlib2==0.6.0 cryptography==2.7 cycler==0.10.0 Cython==0.29.13 cytoolz==0.10.0 dask==1.0.0 datashape==0.5.4 decorator==4.4.0 defusedxml==0.6.0 descartes==1.1.0 dill==0.3.1.1 distributed==1.28.1 docutils==0.15.2 entrypoints==0.3 et-xmlfile==1.0.1 fa2==0.3.5 fastcache==1.1.0 fenics-dijitso==2019.1.0 fenics-ffc==2019.1.0.post0 fenics-fiat==2019.1.0 fenics-ufl==2019.1.0 filelock==3.0.12 Flask==1.1.1 Flask-Cors==3.0.8 frozendict==1.2 fsspec==0.5.2 future==0.18.2 get-version==2.1 gevent==1.4.0 glob2==0.7 gmpy2==2.0.8 greenlet==0.4.15 h5py==2.10.0 HeapDict==1.0.1 html5lib==1.0.1 idna==2.8 imageio==2.6.0 imagesize==1.1.0 importlib-metadata==0.23 interlap==0.2.6 ipykernel==5.1.2 ipyparallel==6.2.4 ipython==7.8.0 ipython-genutils==0.2.0 ipywidgets==7.5.1 isort==4.3.21 itsdangerous==1.1.0 jdcal==1.4.1 jedi==0.15.1 jeepney==0.4.1 Jinja2==2.10.3 joblib==0.13.2 json5==0.8.5 jsonschema==3.0.2 jupyter==1.0.0 jupyter-client==5.3.3 jupyter-console==6.0.0 jupyter-core==4.5.0 jupyterlab==1.1.4 jupyterlab-launcher==0.13.1 jupyterlab-server==1.0.6 jupytext==1.3.2 keyring==18.0.0 kiwisolver==1.1.0 lazy-object-proxy==1.4.2 legacy-api-wrap==1.2 leidenalg==0.8.3 libarchive-c==2.8 lief==0.9.0 llvmlite==0.29.0 locket==0.2.0 loompy==3.0.6 lxml==4.4.1 MarkupSafe==1.1.1 matplotlib==3.2.1 mccabe==0.6.1 metakernel==0.24.3 mistune==0.8.4 mizani==0.7.2 mkl-fft==1.0.14 mkl-random==1.1.0 mkl-service==2.3.0 mock==3.0.5 more-itertools==7.2.0 mpmath==1.1.0 msgpack==0.6.1 MulticoreTSNE==0.1 multipledispatch==0.6.0 multiprocessing-on-dill==3.5.0a4 mygene==3.1.0 natsort==7.0.1 navigator-updater==0.2.1 nb-conda-kernels==2.2.2 nbconvert==5.6.0 nbformat==4.4.0 networkx==2.3 nltk==3.4.5 nose==1.3.7 notebook==6.0.1 numba==0.45.1 numexpr==2.7.0 numpy==1.17.2 numpy-groupies==0+unknown numpydoc==0.9.1 odo==0.5.1 olefile==0.46 opencv-python==4.1.2.30 openpyxl==3.0.0 packaging==19.2 palettable==3.3.0 pandas==0.25.3 pandocfilters==1.4.2 parso==0.5.1 partd==1.0.0 path.py==12.0.1 pathlib2==2.3.5 patsy==0.5.1 pbr==5.4.2 pep8==1.7.1 pexpect==4.7.0 pickleshare==0.7.5 Pillow==6.2.0 pkgconfig==1.5.1 pkginfo==1.5.0.1 plotnine==0.7.1 pluggy==0.13.0 ply==3.11 portalocker==1.5.2 prometheus-client==0.7.1 prompt-toolkit==2.0.10 psutil==5.6.3 ptyprocess==0.6.0 py==1.8.0 pyarrow==0.16.0 pycodestyle==2.5.0 pycosat==0.6.3 pycparser==2.19 pycrypto==2.6.1 pycurl==7.43.0.3 pyflakes==2.1.1 Pygments==2.4.2 pylint==2.4.2 pyodbc==4.0.27 pyOpenSSL==19.0.0 pyparsing==2.4.2 pyrsistent==0.15.4 pysam==0.16.0.1 pyscenic==0.10.4 PySocks==1.7.1 pytest==5.2.1 pytest-arraydiff==0.3 pytest-astropy==0.5.0 pytest-doctestplus==0.4.0 pytest-openfiles==0.4.0 pytest-remotedata==0.3.2 python-dateutil==2.8.0 python-igraph==0.8.3 pytz==2019.3 PyWavelets==1.0.3 PyYAML==5.1.2 pyzmq==18.1.0 QtAwesome==0.6.0 qtconsole==4.5.5 QtPy==1.9.0 requests==2.22.0 rope==0.14.0 ruamel-yaml==0.15.46 scanpy==1.7.0rc1 scikit-image==0.15.0 scikit-learn==0.21.3 scipy==1.6.0 scvelo==0.2.2 seaborn==0.9.0 SecretStorage==3.1.1 Send2Trash==1.5.0 setuptools-scm==3.5.0 simplegeneric==0.8.1 sinfo==0.3.1 singledispatch==3.4.0.3 six==1.12.0 snowballstemmer==2.0.0 sortedcollections==1.1.2 sortedcontainers==2.1.0 soupsieve==1.9.3 Sphinx==2.2.0 sphinxcontrib-applehelp==1.0.1 sphinxcontrib-devhelp==1.0.1 sphinxcontrib-htmlhelp==1.0.2 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.2 sphinxcontrib-serializinghtml==1.1.3 sphinxcontrib-websupport==1.1.2 spyder==3.3.6 spyder-kernels==0.5.2 SQLAlchemy==1.3.9 statsmodels==0.12.1 stdlib-list==0.8.0 sympy==1.4 tables==3.6.1 tbb==2020.0.133 tblib==1.4.0 terminado==0.8.2 testfixtures==6.10.0 testpath==0.4.2 texttable==1.6.3 toolz==0.10.0 tornado==6.0.3 tqdm==4.36.1 traitlets==4.3.3 typing==3.6.4 umap-learn==0.4.2 unicodecsv==0.14.1 urllib3==1.24.2 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.16.0 widgetsnbextension==3.5.1 wrapt==1.11.2 wurlitzer==1.0.3 xarray==0.14.1 xlrd==1.2.0 XlsxWriter==1.2.1 xlwt==1.3.0 zict==1.0.0 zipp==0.6.0

[cflerin]

Thanks for the info. I would recommend:

• Update to pySCENIC 0.11.0 (there are package updates that could resolve this issue)
• If the new version still doesn't help, you could try the multiprocessing version instead of using dask. You can do this by using the --mode "custom_multiprocessing" option in the CLI command.

(similar to #142)

[willrosenow]

Thank you @cflerin !

I'll try both of those suggestions.

[mingshuang1]

Hi! Are your problems resolved? I am facing the same issue now but neither methods work for me