AstreChen commented 4 years ago

Describe the bug Hi, I got a ValueError when runninig "pyscenic ctx". I follow the Nature Protocol and download the ranking database in feather format. The list of TF I used is SCENICprotocol/example/allTFs_hg38.txt, and the gene name in my matrix is also Gene Symbol. I found that some TFs in the adj.csv are not in the feather files (colnames), but after I excluded them, I still got the error. Could you please help me?

Steps to reproduce the behavior

Command run when the error occurred:


pyscenic grn $f_loom_path SCENICprotocol/example/allTFs_hg38.txt-o adj.csv --num_workers 20 --seed 777

pyscenic ctx ./scenic_results/adj.csv /BGFS1/home/aic/tools/scenic/resourc es/hg38refseq-r8010kb_up_and_down_tss.mc9nr.feather \

--annotations_fname ~/tools/scenic/resources/motifs-v9-nr.hgnc-m0.001-o0.0.tbl \ --expression_mtx_fname /BGFS1/home/aic/projects/FTE/submission04/results/20200721_cancerA/CD4Treg_cancerA_sub13467_clustered_annotated_C14.loom \ --mode "dask_multiprocessing" \ --output ./scenic_results/reg.csv \ --num_workers 20 \ --mask_dropouts

2020-08-11 10:13:20,532 - pyscenic.cli.pyscenic - INFO - Creating modules.

2020-08-11 10:13:22,680 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2020-08-11 10:13:29,778 - pyscenic.utils - INFO - Calculating Pearson correlations.

2020-08-11 10:13:29,778 - pyscenic.utils - WARNING - Note on correlation calculation: the default behaviour for calculating the correlations has changed after pySCENIC verion 0.9.16. Previously, the default was to calculate the correlation between a TF and target gene using only cells with non-zero expression values (mask_dropouts=True). The current default is now to use all cells to match the behavior of the R verision of SCENIC. The original settings can be retained by setting 'rho_mask_dropouts=True' in the modules_from_adjacencies function, or '--mask_dropouts' from the CLI. Dropout masking is currently set to [True].

/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/pyscenic/utils.py:138: RuntimeWarning: invalid value encountered in greater regulations = (rhos > rho_threshold).astype(int) - (rhos < -rho_threshold).astype(int) /BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/pyscenic/utils.py:138: RuntimeWarning: invalid value encountered in less regulations = (rhos > rho_threshold).astype(int) - (rhos < -rho_threshold).astype(int)

2020-08-11 10:19:55,129 - pyscenic.utils - INFO - Creating modules.

2020-08-11 10:20:50,275 - pyscenic.cli.pyscenic - INFO - Loading databases.

2020-08-11 10:20:50,276 - pyscenic.cli.pyscenic - INFO - Calculating regulons. [ ] | 0% Completed | 46.6s Traceback (most recent call last): File "/BGFS1/home/aic/.conda/envs/sc/bin/pyscenic", line 8, in sys.exit(main()) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/pyscenic/cli/pyscenic.py", line 420, in main args.func(args) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/pyscenic/cli/pyscenic.py", line 159, in prune_targets_command df_motifs = calc_func(dbs, modules, motif_annotations_fname, File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/pyscenic/prune.py", line 349, in prune2df return _distributed_calc(rnkdbs, modules, motif_annotations_fname, transformation_func, aggregation_func, File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/pyscenic/prune.py", line 300, in _distributed_calc return create_graph().compute(scheduler='processes', num_workers=num_workers if num_workers else cpu_count()) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/base.py", line 166, in compute (result,) = compute(self, traverse=False, kwargs) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/base.py", line 437, in compute results = schedule(dsk, keys, kwargs) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/multiprocessing.py", line 212, in get result = get_async( File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/local.py", line 486, in get_async raise_exception(exc, tb) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/local.py", line 316, in reraise raise exc File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/local.py", line 222, in execute_task result = _execute_task(task, data) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task return func(*args2) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/dataframe/utils.py", line 653, in check_meta check_matching_columns(meta, x) File "/BGFS1/home/aic/.conda/envs/sc/lib/python3.8/site-packages/dask/dataframe/utils.py", line 674, in check_matching_columns raise ValueError( ValueError: The columns in the computed data do not match the columns in the provided metadata Extra: [] Missing: []


**Please complete the following information:**
- pySCENIC version: 0.10.0
- Installation method: Pip
- Run environment:  HPC
- OS: (Ubuntu 7.4.0-1ubuntu1~18.04.1))
- Package versions: [obtain using `pip freeze`, `conda list`, or skip this if using Docker/Singularity]:
<!-- Put your package version list in this code block (if applicable, else delete the block): -->

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_llvm conda-forge anndata 0.7.1 pypi_0 pypi arboreto 0.1.5 pypi_0 pypi attrs 19.3.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge blas 1.0 openblas bleach 3.1.5 pyh9f0ad1d_0 conda-forge blosc 1.18.1 he1b5a44_0 conda-forge boltons 20.2.0 pypi_0 pypi bzip2 1.0.8 h516909a_2 conda-forge ca-certificates 2020.6.24 0 cairo 1.16.0 hcf35c78_1003 conda-forge certifi 2020.6.20 py38_0 cffi 1.14.0 pypi_0 pypi chardet 3.0.4 pypi_0 pypi click 7.1.2 pypi_0 pypi cloudpickle 1.5.0 pypi_0 pypi cmake 3.17.0 h28c56e5_0 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.10.1 pypi_0 pypi dask 2.11.0 pypi_0 pypi dbus 1.13.6 he372182_0 conda-forge decorator 4.4.2 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge dill 0.3.2 pypi_0 pypi distributed 2.11.0 pypi_0 pypi entrypoints 0.3 py38h32f6830_1001 conda-forge expat 2.2.9 he1b5a44_2 conda-forge fa2 0.3.5 pypi_0 pypi fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.1 he06d7ca_0 conda-forge frozendict 1.2 pypi_0 pypi fsspec 0.8.0 pypi_0 pypi get-version 2.1 pypi_0 pypi gettext 0.19.8.1 hc5be6a0_1002 conda-forge glib 2.64.2 h6f030ca_0 conda-forge gmp 6.2.0 he1b5a44_2 conda-forge gprofiler-official 1.0.0 pypi_0 pypi gsl 2.6 h294904e_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_2 conda-forge gstreamer 1.14.5 h36ae1b5_2 conda-forge h5py 2.10.0 nompi_py38h513d04c_102 conda-forge hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge heapdict 1.0.1 pypi_0 pypi icu 64.2 he1b5a44_1 conda-forge idna 2.9 pypi_0 pypi igraph 0.7.1 h9e3b1fc_1007 conda-forge importlib-metadata 1.6.0 py38h32f6830_0 conda-forge importlib_metadata 1.6.0 0 conda-forge interlap 0.2.6 pypi_0 pypi ipykernel 5.2.1 py38h23f93f0_0 conda-forge ipython 7.14.0 py38h32f6830_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 py_0 conda-forge jedi 0.17.0 py38h32f6830_0 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge joblib 0.14.1 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge jsonschema 3.2.0 py38h32f6830_1 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter_client 6.1.3 py_0 conda-forge jupyter_console 6.1.0 py_1 conda-forge jupyter_core 4.6.3 py38h32f6830_1 conda-forge kiwisolver 1.2.0 py38hbf85e49_0 conda-forge krb5 1.17.1 h2fd8d38_0 conda-forge ld_impl_linux-64 2.34 h53a641e_0 conda-forge legacy-api-wrap 1.2 pypi_0 pypi libblas 3.8.0 16_openblas conda-forge libcblas 3.8.0 16_openblas conda-forge libclang 9.0.1 default_hde54327_0 conda-forge libcurl 7.69.1 hf7181ac_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgfortran-ng 7.3.0 hdf63c60_5 conda-forge libiconv 1.15 h516909a_1006 conda-forge liblapack 3.8.0 16_openblas conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libllvm9 9.0.1 he513fc3_1 conda-forge libopenblas 0.3.9 h5ec1e0e_0 conda-forge libpng 1.6.37 hed695b0_1 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.34.0 h516909a_0 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxkbcommon 0.10.0 he1b5a44_0 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvm-openmp 10.0.0 hc9558a2_0 conda-forge llvmlite 0.31.0 py38h4f45e52_1 conda-forge loompy 3.0.6 pypi_0 pypi louvain 0.6.1 py38he1b5a44_2 conda-forge lzo 2.10 h14c3975_1000 conda-forge markupsafe 1.1.1 py38h1e0a361_1 conda-forge matplotlib-base 3.2.1 py38h2af1d28_0 conda-forge mistune 0.8.4 py38h1e0a361_1001 conda-forge mock 4.0.2 py38h32f6830_0 conda-forge msgpack 1.0.0 pypi_0 pypi multicoretsne 0.1 pypi_0 pypi multiprocessing-on-dill 3.5.0a4 pypi_0 pypi natsort 7.0.1 py_0 conda-forge nbconvert 5.6.1 py38h32f6830_1 conda-forge nbformat 5.0.6 py_0 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge networkx 2.4 py_1 conda-forge notebook 6.0.3 py38_0 conda-forge nspr 4.25 he1b5a44_0 conda-forge nss 3.47 he751ad9_0 conda-forge numba 0.48.0 py38hb3f55d8_0 conda-forge numexpr 2.7.1 py38hcb8c335_1 conda-forge numpy 1.18.4 py38h8854b6b_0 conda-forge numpy-groupies 0+unknown pypi_0 pypi openssl 1.1.1g h7b6447c_0 packaging 20.1 py_0 conda-forge pandas 0.25.3 pypi_0 pypi pandoc 2.9.2.1 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.7.0 pyh9f0ad1d_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pexpect 4.8.0 py38h32f6830_1 conda-forge pickleshare 0.7.5 py38h32f6830_1001 conda-forge pip 20.1 pyh9f0ad1d_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt-toolkit 3.0.5 py_0 conda-forge prompt_toolkit 3.0.5 0 conda-forge psutil 5.7.2 pypi_0 pypi pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pyarrow 0.16.0 pypi_0 pypi pycairo 1.19.1 py38h323dad1_3 conda-forge pycparser 2.20 pypi_0 pypi pygments 2.6.1 py_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyqt 5.12.3 py38ha8c2ead_3 conda-forge pyqt5-sip 4.19.18 pypi_0 pypi pyqtchart 5.12 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pyrsistent 0.16.0 py38h1e0a361_0 conda-forge pyscenic 0.10.0 pypi_0 pypi pytables 3.6.1 py38h9f153d1_1 conda-forge python 3.8.2 he5300dc_7_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python-igraph 0.8.2 py38hb721cf7_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytz 2020.1 pyh9f0ad1d_0 conda-forge pyyaml 5.3.1 pypi_0 pypi pyzmq 19.0.0 py38ha71036d_1 conda-forge qt 5.12.5 hd8c4c69_1 conda-forge qtconsole 4.7.3 pyh9f0ad1d_0 conda-forge qtpy 1.9.0 py_0 conda-forge readline 8.0 hf8c457e_0 conda-forge requests 2.23.0 pypi_0 pypi rhash 1.3.6 h14c3975_1001 conda-forge scanpy 1.4.6 pypi_0 pypi schpf 0.3.0 pypi_0 pypi scikit-learn 0.23.1 py38h7ea95a0_0 scipy 1.4.1 py38h18bccfc_3 conda-forge seaborn 0.10.1 py_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 46.1.3 py38h32f6830_0 conda-forge setuptools-scm 3.5.0 pypi_0 pypi simplegeneric 0.8.1 py_1 conda-forge six 1.14.0 py_1 conda-forge sortedcontainers 2.2.2 pypi_0 pypi sqlite 3.30.1 hcee41ef_0 conda-forge statsmodels 0.11.1 py38h7b6447c_0 tbb 2020.0.133 pypi_0 pypi tblib 1.7.0 pypi_0 pypi terminado 0.8.3 py38h32f6830_1 conda-forge testpath 0.4.4 py_0 conda-forge texttable 1.6.2 py_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 tk 8.6.10 hed695b0_0 conda-forge toolz 0.10.0 pypi_0 pypi tornado 6.0.4 py38h1e0a361_1 conda-forge tqdm 4.46.0 pyh9f0ad1d_0 conda-forge traitlets 4.3.3 py38h32f6830_1 conda-forge tzlocal 2.0.0 py_0 conda-forge umap-learn 0.4.2 pypi_0 pypi urllib3 1.25.9 pypi_0 pypi wcwidth 0.1.9 pyh9f0ad1d_0 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.34.2 py_1 conda-forge widgetsnbextension 3.5.1 py38_0 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.5 h516909a_0 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 2.0.0 pypi_0 pypi zipp 3.1.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge

SBata commented 4 years ago

same error here when pruning the data withprune2df()

Byronxy commented 4 years ago

I got the same error! Could anyone help?

cflerin commented 4 years ago

Can you try reinstalling it with the latest pySCENIC version? Via pip:

pip install --force-reinstall pyscenic==0.10.3

Using --force-reinstall will also downgrade your dask/distributed packages, which could be causing this issue.

Manikgarg commented 4 years ago

Can you try reinstalling it with the latest pySCENIC version? Via pip:
pip install --force-reinstall pyscenic==0.10.3
Using --force-reinstall will also downgrade your dask/distributed packages, which could be causing this issue.

This seemed to have worked for me (the code is still running but I am past the point where I was getting this error). Will keep you updated in case I encounter this issue again. Thanks!

sunyolo commented 4 years ago

same error here when pruning the data withprune2df()

Have you solved this problem? I also get the error when using prune2df() in jupyter notebook

Tianqi-Ma commented 4 years ago

I have this same issue as well. My pySCENIC is already 0.10.3.

saguilarfer commented 4 years ago

Same issue, also pySCENIC is already 0.10.3

Coke-Zhang commented 3 years ago

Yeah. same issue, also pySCENIC v0.10.3 based on win10 and ubuntu 18 when using prune2df().

GuiSeSanz commented 3 years ago

At the moment, I'm trying to run it with the last version, Single-CEll regulatory Network Inference and Clustering (0.10.4) And i got the same error....

rosanababu commented 3 years ago

I'm getting following error while running pyscenic ctx - AttributeError: module 'numba' has no attribute 'jitclass', please also see below pip freeze to see the versions I'm using. Kindly help how to proceed.

pyscenic ctx /pyScenic/SCENIC/Figures/Dec8/PBMC/2adjacencies_PBMCnew.tsv /pyScenic/hg38refseq-r8010kb_up_and_down_tss.mc9nr.feather --annotations_fname /pyScenic/motifs-v9-nr.hgnc-m0.001-o0.0.tbl --expression_mtx_fname /pyScenic/SCENIC/Figures/Dec8/PBMC/1_PBMCnew_transpose.csv --mode "dask_multiprocessing" --output /pyScenic/SCENIC/Figures/Dec8/PBMC/3regulons_PBMCnew.csv --num_workers 24 --mask_dropouts

/t1-home/molimm/rottakan/pyscenicvenv/lib/python3.8/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. data = yaml.load(f.read()) or {} Traceback (most recent call last): File "/t1-home/molimm/rottakan/pyscenicvenv/bin/pyscenic", line 6, in from pyscenic.cli.pyscenic import main File "/t1-home/molimm/rottakan/pyscenicvenv/lib/python3.8/site-packages/pyscenic/cli/pyscenic.py", line 19, in from pyscenic.utils import modules_from_adjacencies File "/t1-home/molimm/rottakan/pyscenicvenv/lib/python3.8/site-packages/pyscenic/utils.py", line 6, in from .math import masked_rho4pairs File "/t1-home/molimm/rottakan/pyscenicvenv/lib/python3.8/site-packages/pyscenic/math.py", line 5, in from numba import * AttributeError: module 'numba' has no attribute 'jitclass'

pip freeze adjustText==0.7.3 anaconda==0.0.1.1 anndata==0.7.4 arboreto==0.1.5 attrs==20.3.0 backcall==0.2.0 boltons==20.2.1 certifi==2020.6.20 click==7.1.2 cloudpickle==1.6.0 cycler==0.10.0 cytoolz==0.11.0 dask==1.0.0 decorator==4.4.2 dill==0.3.3 distributed==1.28.1 frozendict==1.2 fsspec==0.8.3 get-version==2.1 h5py==3.1.0 HeapDict==1.0.1 interlap==0.2.7 ipython==7.18.1 ipython-genutils==0.2.0 jedi==0.17.2 joblib==0.17.0 kiwisolver==1.2.0 legacy-api-wrap==1.2 llvmlite==0.35.0 locket==0.2.0 loompy==3.0.6 louvain==0.6.1 matplotlib==3.3.2 msgpack==1.0.0 multiprocessing-on-dill==3.5.0a4 natsort==7.0.1 networkx==2.5 numba==0.52.0 numexpr==2.7.1 numpy==1.19.4 numpy-groupies==0.9.13 packaging==20.4 pandas==0.25.3 parso==0.7.1 partd==1.1.0 patsy==0.5.1 pexpect==4.8.0 pickleshare==0.7.5 Pillow==7.2.0 prompt-toolkit==3.0.7 psutil==5.7.3 ptyprocess==0.6.0 pyarrow==0.16.0 Pygments==2.7.1 pyparsing==2.4.7 pyscenic==0.10.3 python-dateutil==2.8.1 python-igraph==0.8.2 pytz==2020.4 PyYAML==5.3.1 scanpy==1.4.6 scikit-learn==0.23.2 scipy==1.5.4 scitools3==1.0 seaborn==0.11.0 setuptools-scm==4.1.2 six==1.15.0 sortedcontainers==2.3.0 statsmodels==0.12.0 tables==3.6.1 tblib==1.7.0 texttable==1.6.3 threadpoolctl==2.1.0 toolz==0.11.1 tornado==6.1 tqdm==4.54.1 traitlets==5.0.4 umap-learn==0.4.6 wcwidth==0.2.5 zict==2.0.0

grimwoo commented 3 years ago

I have the same issue as rosana08. module 'numba' has no attribute 'jitclass'

rosanababu commented 3 years ago

I have the same issue as rosana08. module 'numba' has no attribute 'jitclass'

I tried downgrading numba, try "pip install numba==0.51.2", this error will be solved. But now I'm getting the same error as others in this thread, "pyscenic ctx - ValueError: The columns in the computed data do not match the columns in the provided metadata ". I have already tried with "pip install --force-reinstall pyscenic==0.10.3" as it resolved my issue two months before and I could run it, but this time it's showing error. I have also tried "pip install --force-reinstall pyscenic==0.10.4", both are giving same errors. First "module 'numba' has no attribute 'jitclass' and when I downgrade numba, the following error "pyscenic ctx - ValueError: The columns in the computed data do not match the columns in the provided metadata".

pawelterentiev commented 3 years ago

Hi everyone, Just curious if there are any updates on this issue? I've spent whole day trying to make things working, but facing all the problems described above? Thank.

rosanababu commented 3 years ago

Hi everyone, Just curious if there are any updates on this issue? I've spent whole day trying to make things working, but facing all the problems described above? Thank.

Hi, Working fine for me now. I think the issue is that pyscenic install gets versions of pandas, dask and distributed that it's not actually compatible with and they need to be over-written after the pyscenic install. So, my order was: pandas -> pyscenic -> fsspec -> dask -> distributed -> numba. Please try below steps for grn steps and create another virtual environment for ctx and aucell steps. I used two different virtual environments, one to run grn step and another for ctx and aucell.

Commands worked for grn step:

load python

module load python-base

create python virtual environment

python -m venv pyscenicvenv

activate newly created environment

source /home/user/pyscenicvenv/bin/activate

create and save a requirements.txt file in input-output folder with below requirements

arboreto==0.1.5 attrs==20.2.0 boltons==20.2.1 click==7.1.2 cloudpickle==1.6.0 cytoolz==0.11.0 dask==1.0.0 decorator==4.4.2 dill==0.3.2 distributed==2.11.0 frozendict==1.2 fsspec==0.8.3 h5py==2.10.0 HeapDict==1.0.1 interlap==0.2.6 joblib==0.16.0 llvmlite==0.34.0 locket==0.2.0 loompy==3.0.6 msgpack==1.0.0 multiprocessing-on-dill==3.5.0a4 networkx==2.5 numba==0.51.2 numpy==1.19.2 numpy-groupies==0.9.13 pandas==0.25.3 partd==1.1.0 psutil==5.7.2 pyarrow==0.16.0 pyscenic==0.10.0 python-dateutil==2.8.1 pytz==2020.1 pyYAML==5.3.1 scikit-learn==0.23.2 scipy==1.5.2 six==1.15.0 sortedcontainers==2.2.2 tblib==1.7.0 threadpoolctl==2.1.0 toolz==0.11.1 tornado==6.0.3 tqdm==4.50.0 umap-learn==0.4.6 zict==2.0.0

install requirements

pip install -r requirements.txt

After "pip install -r requirements.txt" again install below in order before running GRN

Order for install: pandas -> pyscenic -> fsspec -> dask -> distributed

pip install pandas==0.25.3 pip install pyscenic==0.10.0 pip install fsspec==0.8.3 pip install dask==1.0.0 pip install distributed==2.11.0

Submit job in that env

pyscenic grn --num_workers 24 --output /t1-data/user/2adjacencies_PBMCnew.tsv --method grnboost2 /t1-data/user/3_PBMCnew_transpose.csv /t1-home/pyScenic/hs_hgnc_tfs.txt ##########################################################

for ctx and aucell steps

create another virtual environment and install requirements.txt and then do the following installation in below order to run ctx and aucell steps.. This worked for me.

pip install pandas==0.25.3 pip install --force-reinstall pyscenic==0.10.4 pip install fsspec==0.7.4 pip install dask==1.0.0 pip install distributed==1.28.1 pip install numba==0.50.0

Hope that helps to solve issue

piyushjo15 commented 3 years ago

Hi All,

I have found a work around that has worked for me on my HPC PBS system. I realized two things are not working. With the updated daskand distributed, the ctxand aucell doesn't work And for grn step I had to use the updated dask and distributed. Also, distrubted==1.28.1 didn't work for me, and my code was getting stuck as IDLE.

So I creade two conda enviroments.

For pyscenic grn step following worked with python 3.8. The solution mentioned in #147. Install pyscenic (recent version) with pip install pyscenic==0.10.4 then the following

python -m pip install 'fsspec>=0.3.3'
python -m pip install dask[dataframe] --upgrade
pip install distributed -U

For 'ctx' and 'aucell' steps, I installed pyscenic==0.10.4 again in another conda environment with same python version, but adjusted version of following libraries

pip install dask==1.0.0
pip install distributed==1.27.1 
pip install numba==0.49.0

I hope this could help some people!

pawelterentiev commented 3 years ago

Hey, rosana08 and piyushjo15, Thanks a lot for prompt reply. Your comments were extemly helpful, much appreciated! All is working perfectly on my side now. That definitely looks like versioning thingy. Best

aertslab / pySCENIC

pyscenic ctx - ValueError: The columns in the computed data do not match the columns in the provided metadata #198

Name Version Build Channel

load python

create python virtual environment

activate newly created environment

create and save a requirements.txt file in input-output folder with below requirements

install requirements

After "pip install -r requirements.txt" again install below in order before running GRN

Order for install: pandas -> pyscenic -> fsspec -> dask -> distributed

Submit job in that env

for ctx and aucell steps