search

aertslab / scenicplus

SCENIC+ is a python package to build gene regulatory networks (GRNs) using combined or separate single-cell gene expression (scRNA-seq) and single-cell chromatin accessibility (scATAC-seq) data.
Other
186 stars 29 forks source link

Numba Issue #256

Closed mayowaakinwale closed 1 year ago

mayowaakinwale commented 1 year ago

What type of problem are you experiencing and which function is you problem related too Hi, Thanks for developing this tool. I used an earlier beta version last year fine but am having an issue with the last step in the pipeline for the release at the date of publication. 

from scenicplus.wrappers.run_scenicplus import run_scenicplus
try:
    # sys.stderr = open(os.devnull, "w")  # silence stderr
    run_scenicplus(
        scplus_obj = scplus_obj,
        variable = ['GEX_celltype'],
        species = 'hsapiens',
        assembly = 'hg38',
        tf_file = '/home/mayowa/tf_db_fpath/utoronto_human_tfs_v_1.01.txt',
        save_path = os.path.join(work_dir, 'scenicplus'),
        biomart_host = biomart_host,
        upstream = [1000, 150000],
        downstream = [1000, 150000],
        calculate_TF_eGRN_correlation = True,
        calculate_DEGs_DARs = True,
        export_to_loom_file = False,
        export_to_UCSC_file = True,
        path_bedToBigBed = '/nfs/turbo/umms-spencejr/SCENIC+_runs/pycisTopic',
        n_cpu = 20,
        _temp_dir = os.path.join(tmp_dir, 'ray_spill'))
except Exception as e:
    dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=-1)
    raise(e)`

Everything seems to run smoothly until towards the end when I get an error for the "Making eGRNs AUC UMAP" part.

Running using 20 cores: 100%|██████████| 29651/29651 [40:05<00:00, 12.32it/s] 2023-11-07 22:13:06,773 GSEA         INFO     Subsetting on adjusted pvalue: 1, minimal NES: 0 and minimal leading edge genes 10
2023-11-07 22:13:16,183 GSEA         INFO     Merging eRegulons
2023-11-07 22:13:19,514 GSEA         INFO     Storing eRegulons in .uns[eRegulons].
2023-11-07 22:15:23,554 SCENIC+_wrapper INFO     Formatting eGRNs
2023-11-07 22:18:37,819 SCENIC+_wrapper INFO     Converting eGRNs to signatures
2023-11-07 22:19:20,384 SCENIC+_wrapper INFO     Calculating eGRNs AUC
2023-11-07 22:19:20,384 SCENIC+_wrapper INFO     Calculating region ranking
2023-11-07 22:53:42,541 SCENIC+_wrapper INFO     Calculating eGRNs region based AUC

2023-11-07 23:02:08,537 SCENIC+_wrapper INFO     Calculating gene ranking
2023-11-07 23:02:50,490 SCENIC+_wrapper INFO     Calculating eGRNs gene based AUC
2023-11-07 23:04:01,035 SCENIC+_wrapper INFO     Calculating TF-eGRNs AUC correlation
2023-11-07 23:05:13,500 SCENIC+_wrapper INFO     Binarizing eGRNs AUC
2023-11-07 23:26:06,662 SCENIC+_wrapper INFO     Making eGRNs AUC UMAP
Traceback (most recent call last):
  File "scplusrun_v1.0.0_6534.py", line 57, in <module>
    raise(e)
  File "scplusrun_v1.0.0_6534.py", line 38, in <module>
    run_scenicplus(
  File "/home/mayowa/test/scenicplus/src/scenicplus/wrappers/run_scenicplus.py", line 290, in run_scenicplus
    run_eRegulons_umap(scplus_obj,
  File "/home/mayowa/test/scenicplus/src/scenicplus/dimensionality_reduction.py", line 294, in run_eRegulons_umap
    embedding = reducer.fit_transform(data_mat)
  File "/home/mayowa/.conda/envs/test/lib/python3.8/site-packages/umap/umap_.py", line 2887, in fit_transform
    self.fit(X, y, force_all_finite)
  File "/home/mayowa/.conda/envs/test/lib/python3.8/site-packages/umap/umap_.py", line 2608, in fit
    ) = nearest_neighbors(
  File "/home/mayowa/.conda/envs/test/lib/python3.8/site-packages/umap/umap_.py", line 329, in nearest_neighbors
    knn_search_index = NNDescent(
  File "/home/mayowa/.conda/envs/test/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 921, in __init__
    self._neighbor_graph = nn_descent(
  File "/home/mayowa/.conda/envs/test/lib/python3.8/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/mayowa/.conda/envs/test/lib/python3.8/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'print': Cannot determine Numba type of <class 'function'>

File "../.conda/envs/test/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 252:
def nn_descent_internal_low_memory_parallel(
    <source elided>
        if verbose:
            print("\t", n + 1, " / ", n_iters)
            ^

During: resolving callee type: type(CPUDispatcher(<function nn_descent_internal_low_memory_parallel at 0x1516898dd3a0>))
During: typing of call at /home/mayowa/.conda/envs/test/lib/python3.8/site-packages/pynndescent/pynndescent_.py (358)

During: resolving callee type: type(CPUDispatcher(<function nn_descent_internal_low_memory_parallel at 0x1516898dd3a0>))
During: typing of call at /home/mayowa/.conda/envs/test/lib/python3.8/site-packages/pynndescent/pynndescent_.py (358)


File "../.conda/envs/test/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 358:
def nn_descent(
    <source elided>
    if low_memory:
        nn_descent_internal_low_memory_parallel(
        ^

I'm still kind of new to all this so I apologize if this has already been addressed elsewhere but has anyone else encountered this issue and if so, what did you do to resolve it? I guess it may be a dependency issue with numba—I made a new test conda environment to double check to no avail—and want to ask if there is a different numba version needed to those of you who have successfully used the scenicplus version release at the date of publication?

Version information SCENIC+: 1.0.1.dev4+ge4bdd9f accessible-pygments==0.0.4 adjustText==0.8 aiohttp==3.8.6 aiohttp-cors==0.7.0 aiosignal==1.3.1 alabaster==0.7.13 anndata==0.9.2 annoy==1.17.3 appdirs==1.4.4 arboreto==0.1.6 asttokens==2.4.1 async-timeout==4.0.3 attr==0.3.2 attrs==23.1.0 Babel==2.13.1 backcall==0.2.0 backports.zoneinfo==0.2.1 bbknn==1.6.0 beautifulsoup4==4.12.2 bidict==0.22.1 bioservices==1.11.2 bleach==6.1.0 blessed==1.20.0 blosc2==2.0.0 bokeh==3.1.1 boltons==23.1.1 bs4==0.0.1 cachetools==5.3.2 cattrs==23.1.2 certifi==2023.7.22 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==3.0.0 colorama==0.4.6 colorful==0.5.5 colorlog==6.7.0 comm==0.2.0 contourpy==1.1.1 cryptography==41.0.5 ctxcore==0.2.0 cycler==0.12.1 Cython==0.29.36 cytoolz==0.12.2 dask==2023.5.0 dataclasses-json==0.6.1 debugpy==1.8.0 decorator==5.1.1 defusedxml==0.7.1 dill==0.3.7 distlib==0.3.7 distributed==2023.5.0 docutils==0.18.1 easydev==0.12.1 entrypoints==0.4 et-xmlfile==1.1.0 exceptiongroup==1.1.3 executing==2.0.1 fastjsonschema==2.18.1 fbpca==1.0 filelock==3.13.1 fonttools==4.44.0 frozendict==2.3.8 frozenlist==1.4.0 fsspec==2023.10.0 future==0.18.3 gensim==4.3.2 geosketch==1.2 get-annotations==0.1.2 gevent==23.9.1 globre==0.1.5 google-api-core==2.13.0 google-auth==2.23.4 googleapis-common-protos==1.61.0 gpustat==1.1.1 greenlet==3.0.1 grequests==0.7.0 grpcio==1.59.2 gseapy==0.10.8 h5py==3.10.0 harmonypy==0.0.9 idna==3.4 igraph==0.10.8 imageio==2.32.0 imagesize==1.4.1 importlib-metadata==6.8.0 importlib-resources==6.1.1 interlap==0.2.7 intervaltree==3.1.0 ipykernel==6.26.0 ipympl==0.9.3 ipython==8.12.3 ipython-genutils==0.2.0 ipywidgets==8.1.1 jedi==0.19.1 Jinja2==3.1.2 joblib==1.3.2 jsonpickle==3.0.2 jsonschema==4.19.2 jsonschema-specifications==2023.7.1 jupyter_client==8.6.0 jupyter_core==5.5.0 jupyterlab-pygments==0.2.2 jupyterlab-widgets==3.0.9 kaleido==0.2.1 kiwisolver==1.4.5 lazy_loader==0.3 lda==2.0.0 leidenalg==0.10.1 Levenshtein==0.23.0 llvmlite==0.41.1 locket==1.0.0 loompy==3.0.7 loomxpy @ git+https://github.com/aertslab/LoomXpy@61995ff10940968eac2cee8fe48300ab477a15d0 lxml==4.9.3 lz4==4.3.2 MACS2==2.2.9.1 MarkupSafe==2.1.3 marshmallow==3.20.1 matplotlib==3.7.3 matplotlib-inline==0.1.6 mistune==3.0.2 mizani==0.9.3 msgpack==1.0.7 mudata==0.2.3 multidict==6.0.4 multiprocessing-on-dill==3.5.0a4 mypy-extensions==1.0.0 natsort==8.4.0 nbclient==0.9.0 nbconvert==7.11.0 nbformat==5.9.2 nbsphinx==0.9.3 nbsphinx-link==1.3.0 ncls==0.0.68 nest-asyncio==1.5.8 networkx==3.1 numba==0.58.1 numexpr==2.8.6 numpy==1.24.4 numpy-groupies==0.9.22 numpydoc==1.6.0 nvidia-ml-py==12.535.133 opencensus==0.11.3 opencensus-context==0.1.3 openpyxl==3.1.2 packaging==23.2 pandas==1.5.0 pandoc==2.3 pandocfilters==1.5.0 papermill==2.5.0 parso==0.8.3 partd==1.4.1 patsy==0.5.3 pbr==3.1.1 pexpect==4.8.0 pickleshare==0.7.5 Pillow==10.0.1 pkgutil_resolve_name==1.3.10 platformdirs==3.11.0 plotly==5.18.0 plotnine==0.12.4 plumbum==1.8.2 ply==3.11 polars==0.19.12 progressbar2==4.2.0 prometheus-client==0.18.0 prompt-toolkit==3.0.39 protobuf==4.25.0 psutil==5.9.6 ptyprocess==0.7.0 pure-eval==0.2.2 py-cpuinfo==9.0.0 py-spy==0.3.14 pyarrow==14.0.0 pyasn1==0.5.0 pyasn1-modules==0.3.0 pybedtools==0.9.1 pyBigWig==0.3.22 pybiomart==0.2.0 pycistarget @ git+https://github.com/aertslab/pycistarget@3fde1ce5e16241f00078bee920d8a805ccbf2360 pycisTopic @ git+https://github.com/aertslab/pycisTopic@8955c76698c7a8a9086b4a58e9fb57f7f7335bf2 pycparser==2.21 pydantic==1.10.13 pydata-sphinx-theme==0.14.3 pyfasta==0.5.2 pygam==0.9.0 Pygments==2.16.1 pynndescent==0.5.10 pyOpenSSL==23.3.0 pyparsing==3.1.1 pyranges==0.0.127 pysam==0.22.0 pyscenic @ git+https://github.com/aertslab/pySCENIC@31d51a1625f12fb3c6e92bc48ecc9d401524c22a python-dateutil==2.8.2 python-igraph==0.10.8 python-Levenshtein==0.23.0 python-utils==3.8.1 pytz==2023.3.post1 pyvis==0.3.2 PyWavelets==1.4.1 PyYAML==6.0.1 pyzmq==25.1.1 rapidfuzz==3.5.2 ray==2.8.0 referencing==0.30.2 requests==2.31.0 requests-cache==1.1.0 rpds-py==0.12.0 rsa==4.9 scanorama==1.7.4 scanpy==1.9.6 -e git+https://github.com/aertslab/scenicplus@e4bdd9f5b7fea1a43d5cb55e2ccb3c221fe7d279#egg=scenicplus scikit-image==0.21.0 scikit-learn==1.3.2 scipy==1.10.1 scrublet==0.2.3 seaborn==0.12.2 session-info==1.0.0 six==1.16.0 smart-open==6.4.0 snowballstemmer==2.2.0 sorted-nearest==0.0.39 sortedcontainers==2.4.0 soupsieve==2.5 Sphinx==6.2.1 sphinx-book-theme==1.0.1 sphinx-rtd-theme==1.3.0 sphinxcontrib-applehelp==1.0.4 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.1 sphinxcontrib-jquery==4.1 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 stack-data==0.6.3 statistics==1.0.3.5 statsmodels==0.14.0 stdlib-list==0.9.0 suds-community==1.1.2 tables==3.8.0 tabulate==0.9.0 tbb==2021.10.0 tblib==3.0.0 tenacity==8.2.3 texttable==1.7.0 threadpoolctl==3.2.0 tifffile==2023.7.10 tinycss2==1.2.1 tmtoolkit==0.12.0 tomli==2.0.1 toolz==0.12.0 tornado==6.3.3 tqdm==4.66.1 traitlets==5.13.0 tspex==0.6.3 typing==3.7.4.3 typing-inspect==0.9.0 typing_extensions==4.8.0 umap-learn==0.5.4 url-normalize==1.4.3 urllib3==2.0.7 virtualenv==20.21.0 wcwidth==0.2.9 webencodings==0.5.1 widgetsnbextension==4.0.9 wrapt==1.15.0 xlrd==2.0.1 xmltodict==0.13.0 xyzservices==2023.10.1 yarl==1.9.2 zict==3.0.0 zipp==3.17.0 zope.event==5.0 zope.interface==6.1

SeppeDeWinter commented 1 year ago

@mayowaakinwale

I will close this issue because it's duplicate to the one mentioned above.

All the best,

Seppe