search

KrishnaswamyLab / phateR

PHATE dimensionality reduction method implemented in R
GNU General Public License v2.0
77 stars 9 forks source link

ModuleNotFoundError: No module named 'pyparsing' #56

Closed reberya closed 3 years ago

reberya commented 3 years ago

Describe the bug Hello,

I am trying to use Phate on my univeristy's HPC and have been unable to run phate to completion on any matrix due to an error with the associated pyparsing module. To address this I have tried to pip install --user pyparsing however I am informed that the requirement is already met:

[user@node ~]$ pip install --user pyparsing
Requirement already satisfied: pyparsing in /sw/arcts/centos7/python3.8-anaconda/2020.07/lib/python3.8/site-packages (2.4.7)

To Reproduce phateR::check_pyphate_version() is sufficient to trigger the error - output can be seen below

Expected behavior Phate to run to completion and return its output.

Actual behavior

following counts_phate <- phate(counts_matrix) the following error returns:

Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'pyparsing'
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'pyparsing'
Error: $ operator is invalid for atomic vectors

System information:

Output of phate.__version__:

[usr@node ~]$ python -c 'import phate; print(phate.__version__)'
1.0.7
[usr@node ~]$

Output of pd.show_versions():

``` INSTALLED VERSIONS ------------------ commit : f2c8480af2f25efdbd803218b9d87980f416563e python : 3.8.3.final.0 python-bits : 64 OS : Linux OS-release : 3.10.0-1160.6.1.el7.x86_64 Version : #1 SMP Tue Nov 17 13:59:11 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 1.2.3 numpy : 1.20.1 pytz : 2021.1 dateutil : 2.8.1 pip : 20.1.1 setuptools : 49.2.0.post20200714 Cython : 0.29.21 pytest : 5.4.3 hypothesis : None sphinx : 3.1.2 blosc : None feather : None xlsxwriter : 1.2.9 lxml.etree : 4.5.2 html5lib : 1.1 pymysql : None psycopg2 : None jinja2 : 2.11.2 IPython : 7.16.1 pandas_datareader: None bs4 : 4.9.1 bottleneck : 1.3.2 fsspec : 0.7.4 fastparquet : None gcsfs : None matplotlib : 3.3.4 numexpr : 2.7.1 odfpy : None openpyxl : 3.0.4 pandas_gbq : None pyarrow : None pyxlsb : None s3fs : None scipy : 1.6.1 sqlalchemy : 1.3.18 tables : 3.6.1 tabulate : None xarray : None xlrd : 1.2.0 xlwt : 1.3.0 numba : 0.50.1 ```

Output of sessionInfo():

``` R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out" Copyright (C) 2020 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > library(phateR); sessionInfo() Loading required package: Matrix R version 4.0.3 (2020-10-10) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core) Matrix products: default BLAS: /sw/arcts/centos7/stacks/gcc/8.2.0/R/4.0.3/lib64/R/lib/libRblas.so LAPACK: /sw/arcts/centos7/stacks/gcc/8.2.0/R/4.0.3/lib64/R/lib/libRlapack.so locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] phateR_1.0.7 Matrix_1.2-18 loaded via a namespace (and not attached): [1] Rcpp_1.0.6 magrittr_2.0.1 tidyselect_1.1.0 munsell_0.5.0 [5] colorspace_2.0-0 lattice_0.20-41 R6_2.5.0 rlang_0.4.10 [9] fastmap_1.1.0 fansi_0.4.2 dplyr_1.0.4 grid_4.0.3 [13] gtable_0.3.0 utf8_1.1.4 DBI_1.1.1 ellipsis_0.3.1 [17] assertthat_0.2.1 tibble_3.0.6 lifecycle_1.0.0 crayon_1.4.1 [21] purrr_0.3.4 ggplot2_3.3.3 vctrs_0.3.6 memoise_2.0.0 [25] glue_1.4.2 cachem_1.0.4 compiler_4.0.3 pillar_1.5.0 [29] generics_0.1.0 scales_1.1.1 jsonlite_1.7.2 reticulate_1.18 [33] pkgconfig_2.0.3 ```

Output of reticulate::py_discover_config(required_module = "phate"):

``` > reticulate::py_discover_config(required_module = "phate") python: /home/usr/miniconda3/envs/r-reticulate/bin/python libpython: /home/usr/miniconda3/envs/r-reticulate/lib/libpython3.8.so pythonhome: /home/usr/miniconda3/envs/r-reticulate:/home/usr/miniconda3/envs/r-reticulate version: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) [GCC 9.3.0] numpy: /home/usr/.local/lib/python3.8/site-packages/numpy numpy_version: 1.20.1 phate: /home/usr/.local/lib/python3.8/site-packages/phate python versions found: /home/usr/miniconda3/envs/r-reticulate/bin/python /sw/arcts/centos7/python3.8-anaconda/2020.07/bin/python3 /usr/bin/python3 /usr/bin/python /home/usr/miniconda3/bin/python ```

Output of phateR::check_pyphate_version():

``` Error loading Python module phate ModuleNotFoundError: No module named 'pyparsing' Detailed traceback: File "/home/usr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/python/rpytools/loader.py", line 19, in _import_hook module = _import( File "/home/usr/.local/lib/python3.8/site-packages/phate/__init__.py", line 3, in from .phate import PHATE File "/home/usr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/python/rpytools/loader.py", line 19, in _import_hook module = _import( File "/home/usr/.local/lib/python3.8/site-packages/phate/phate.py", line 17, in import matplotlib.pyplot as plt File "/home/usr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/python/rpytools/loader.py", line 19, in _import_hook module = _import( File "/home/usr/.local/lib/python3.8/site-packages/matplotlib/__init__.py", line 107, in from . import cbook, rcsetup File "/home/usr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/python/rpytools/loader.py", line 19, in _import_hook module = _import( File "/home/usr/.local/lib/python3.8/site-packages/matplotlib/rcsetup.py", line 28, in from matplotlib.fontconfig_pattern import parse_fontconfig_pattern File "/home/usr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/python/rpytools/loader.py", line 19, in _import_hook module = _import( File "/home/usr/.local/lib/python3.8/site-packages/matplotlib/fontconfig_pattern.py", line 15, in from pyparsing import (Literal, ZeroOrMore, Optional, Regex, StringEnd, File "/home/usr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/python/rpytools/loader.py", line 19, in _import_hook module = _import( Consider removing the 'r-reticulate' environment by running: reticulate::conda_remove('r-reticulate') Error: Error loading Python module phate Execution halted ```

Output of reticulate::py_discover_config(required_module = "phate"):

``` > reticulate::py_discover_config(required_module = "phate") python: /home/usr/miniconda3/envs/r-reticulate/bin/python libpython: /home/usr/miniconda3/envs/r-reticulate/lib/libpython3.8.so pythonhome: /home/usr/miniconda3/envs/r-reticulate:/home/usr/miniconda3/envs/r-reticulate version: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) [GCC 9.3.0] numpy: /home/usr/.local/lib/python3.8/site-packages/numpy numpy_version: 1.20.1 phate: /home/usr/.local/lib/python3.8/site-packages/phate python versions found: /home/usr/miniconda3/envs/r-reticulate/bin/python /sw/arcts/centos7/python3.8-anaconda/2020.07/bin/python3 /usr/bin/python3 /usr/bin/python /home/usr/miniconda3/bin/python ```

Output of reticulate::py_discover_config(required_module = "pyparsing"):

``` > reticulate::py_discover_config(required_module = "pyparsing") python: /sw/arcts/centos7/python3.8-anaconda/2020.07/bin/python3 libpython: /sw/arcts/centos7/python3.8-anaconda/2020.07/lib/libpython3.8.so pythonhome: /sw/arcts/centos7/python3.8-anaconda/2020.07:/sw/arcts/centos7/python3.8-anaconda/2020.07 version: 3.8.3 (default, Jul 2 2020, 16:21:59) [GCC 7.3.0] numpy: /home/usr/.local/lib/python3.8/site-packages/numpy numpy_version: 1.20.1 pyparsing: /sw/arcts/centos7/python3.8-anaconda/2020.07/lib/python3.8/site-packages python versions found: /home/usr/miniconda3/envs/r-reticulate/bin/python /sw/arcts/centos7/python3.8-anaconda/2020.07/bin/python3 /usr/bin/python3 /usr/bin/python /home/usr/miniconda3/bin/python ```
reberya commented 3 years ago

I seem to have solved my own issue. For other's future reference what I did: Looking at the output of reticulate::py_discover_config(required_module = "phate") and reticulate::py_discover_config(required_module = "pyparsing") you can see that the versions of python being used are different. I suspect this is because using an HPC's available modules of python has a different destination than phate which I installed via pip install --user phate.

To address this I specified phate's target destination and reinstalled via: pip install --user --target=/home/usr/.local/lib/python3.8/site-packages/phate --upgrade phate This ensured that all the associated required packages would be installed in the same directory rather than relying on their presence in other python versions present on my system. Following this looking at the output below you can see that both pyparsing and phate have the same location within the system.

Other ways I think could solve this problem (granted I am obviously not that fluent with python/python envs):

Working output of reticulate::py_discover_config(required_module = "pyparsing"):

``` > reticulate::py_discover_config(required_module = "pyparsing") python: /home/usr/miniconda3/envs/r-reticulate/bin/python libpython: /home/usr/miniconda3/envs/r-reticulate/lib/libpython3.8.so pythonhome: /home/usr/miniconda3/envs/r-reticulate:/home/usr/miniconda3/envs/r-reticulate version: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) [GCC 9.3.0] numpy: /home/usr/.local/lib/python3.8/site-packages/numpy numpy_version: 1.20.1 pyparsing: /home/usr/.local/lib/python3.8/site-packages python versions found: /home/usr/miniconda3/envs/r-reticulate/bin/python /home/usr/miniconda3/envs/r-reticulate/bin/python3 /usr/bin/python3 /usr/bin/python /home/usr/miniconda3/bin/python ```

Working output of reticulate::py_discover_config(required_module = "phate")

``` > reticulate::py_discover_config(required_module = "phate") python: /home/usr/miniconda3/envs/r-reticulate/bin/python libpython: /home/usr/miniconda3/envs/r-reticulate/lib/libpython3.8.so pythonhome: /home/usr/miniconda3/envs/r-reticulate:/home/usr/miniconda3/envs/r-reticulate version: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) [GCC 9.3.0] numpy: /home/usr/.local/lib/python3.8/site-packages/numpy numpy_version: 1.20.1 phate: /home/usr/.local/lib/python3.8/site-packages/phate python versions found: /home/usr/miniconda3/envs/r-reticulate/bin/python /home/usr/miniconda3/envs/r-reticulate/bin/python3 /usr/bin/python3 /usr/bin/python /home/usr/miniconda3/bin/python ```

This website helped me solve my issue: https://www.bswen.com/2019/09/others-pip-install-conan-errors-and-exceptions.html