search

rstudio / reticulate

R Interface to Python
https://rstudio.github.io/reticulate
Apache License 2.0
1.68k stars 327 forks source link

module not found when installed with `conda_install()` #838

Open micdonato opened 4 years ago

micdonato commented 4 years ago

Hi all.

I am trying to install loompy and anndata for use with sceasy.

This is my output, from a fresh session. The module installs properly, it seems, but reticulate cannot detect it. Strangely enough, this does not happen for numpy, but only for loompy

This is the code I use:

library(reticulate)

Sys.setenv(PYTHONHOME = "")
Sys.setenv(PYTHONPATH = "")

# reticulate::py_config()
reticulate::use_condaenv('r-reticulate', required = TRUE)
reticulate::conda_list()
reticulate::conda_binary()

reticulate::conda_install(packages = "numpy")
reticulate::py_module_available('numpy') 

reticulate::conda_install(packages = "loompy", channel = 'bioconda')
reticulate::py_module_available('loompy') 
loompy <- reticulate::import('loompy')

reticulate::py_config()
reticulate::conda_list()
reticulate::conda_binary()

Here I am setting the empty environment variables PYTHONHOME and PYTHONPATH but nothing changes if I remove them.

Also, I am forcing the use of r-reticulate as an environment, but again, nothing changes if I remove that.

This is the output:

R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(reticulate)
> 
> Sys.setenv(PYTHONHOME = "")
> Sys.setenv(PYTHONPATH = "")
> 
> # reticulate::py_config()
> reticulate::use_condaenv('r-reticulate', required = TRUE)
> reticulate::conda_list()
          name                                                               python
1   miniconda3                   /local-scratch/users/myuser/miniconda3/bin/python
2 r-reticulate /local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python
> reticulate::conda_binary()
[1] "/local-scratch/users/myuser/miniconda3/bin/conda"
> 
> reticulate::conda_install(packages = "numpy")
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done

## Package Plan ##

  environment location: /local-scratch/users/myuser/miniconda3/envs/r-reticulate

  added / updated specs:
    - python

The following packages will be UPDATED:

  anndata            bioconda/noarch::anndata-0.6.22.post1~ --> conda-forge/linux-64::anndata-0.7.4-py38h32f6830_0
  loompy                       bioconda::loompy-2.0.16-py_0 --> conda-forge::loompy-2.0.17-py_0
  python                          3.6.11-h4d41432_2_cpython --> 3.8.5-h1103e12_7_cpython
  python_abi                                    3.6-1_cp36m --> 3.8-1_cp38
  typing                    bioconda::typing-3.5.2.2-py36_0 --> conda-forge::typing-3.7.4.3-py38h32f6830_0

The following packages will be DOWNGRADED:

  certifi                          2020.6.20-py36h9f0ad1d_0 --> 2020.6.20-py38h32f6830_0
  h5py                        2.10.0-nompi_py36hecadee3_104 --> 2.10.0-nompi_py38hfb01d0b_104
  numpy                               1.19.1-py36h3849536_2 --> 1.19.1-py38hbc27379_2
  pandas                               1.1.1-py36h831f99a_0 --> 1.1.1-py38h950e882_0
  scipy                                1.5.2-py36h3a855aa_0 --> 1.5.2-py38h8c5af15_0
  setuptools                          49.6.0-py36h9f0ad1d_0 --> 49.6.0-py38h32f6830_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

> reticulate::py_module_available('numpy') 
[1] TRUE
> 
> reticulate::conda_install(packages = "loompy", channel = 'bioconda')
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done

## Package Plan ##

  environment location: /local-scratch/users/myuser/miniconda3/envs/r-reticulate

  added / updated specs:
    - loompy
    - python

The following packages will be SUPERSEDED by a higher-priority channel:

  anndata            conda-forge/linux-64::anndata-0.7.4-p~ --> bioconda/noarch::anndata-0.6.22.post1-py_0
  loompy                    conda-forge::loompy-2.0.17-py_0 --> bioconda::loompy-2.0.16-py_0
  typing             conda-forge::typing-3.7.4.3-py38h32f6~ --> bioconda::typing-3.5.2.2-py36_0

The following packages will be DOWNGRADED:

  certifi                          2020.6.20-py38h32f6830_0 --> 2020.6.20-py36h9f0ad1d_0
  h5py                        2.10.0-nompi_py38hfb01d0b_104 --> 2.10.0-nompi_py36hecadee3_104
  numpy                               1.19.1-py38hbc27379_2 --> 1.19.1-py36h3849536_2
  pandas                               1.1.1-py38h950e882_0 --> 1.1.1-py36h831f99a_0
  python                           3.8.5-h1103e12_7_cpython --> 3.6.11-h4d41432_2_cpython
  python_abi                                     3.8-1_cp38 --> 3.6-1_cp36m
  scipy                                1.5.2-py38h8c5af15_0 --> 1.5.2-py36h3a855aa_0
  setuptools                          49.6.0-py38h32f6830_0 --> 49.6.0-py36h9f0ad1d_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
> reticulate::py_module_available('loompy') 
[1] FALSE
> loompy <- reticulate::import('loompy')
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'loompy'
> 
> reticulate::py_config()
python:         /local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python
libpython:      /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/libpython3.8.so[NOT FOUND]
pythonhome:     /local-scratch/users/myuser/miniconda3/envs/r-reticulate:/local-scratch/users/myuser/miniconda3/envs/r-reticulate
version:        3.8.5 | packaged by conda-forge | (default, Aug 29 2020, 01:22:49)  [GCC 7.5.0]
numpy:          /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.8/site-packages/numpy
numpy_version:  1.19.1
numpy:          /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.8/site-packages/numpy

NOTE: Python version was forced by use_python function
> reticulate::conda_list()
          name                                                               python
1   miniconda3                   /local-scratch/users/myuser/miniconda3/bin/python
2 r-reticulate /local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python
> reticulate::conda_binary()
[1] "/local-scratch/users/myuser/miniconda3/bin/conda"
> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblasp-r0.3.3.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.15

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5        rminiconda_0.0.1  compiler_3.6.3    prettyunits_1.1.1 remotes_2.1.1    
 [6] tools_3.6.3       testthat_2.3.2    digest_0.6.25     pkgbuild_1.0.8    pkgload_1.0.2    
[11] jsonlite_1.6.1    evaluate_0.14     memoise_1.1.0     lattice_0.20-38   rlang_0.4.6      
[16] Matrix_1.2-18     cli_2.0.2         rstudioapi_0.11   yaml_2.2.1        xfun_0.13        
[21] withr_2.2.0       knitr_1.28        fs_1.4.1          desc_1.2.0        devtools_2.3.0   
[26] rprojroot_1.3-2   grid_3.6.3        glue_1.4.2        R6_2.4.1          processx_3.4.2   
[31] fansi_0.4.1       rmarkdown_2.3     sessioninfo_1.1.1 callr_3.4.3       magrittr_1.5     
[36] usethis_1.6.1     backports_1.1.7   ps_1.3.3          ellipsis_0.3.1    htmltools_0.4.0  
[41] assertthat_0.2.1  crayon_1.3.4     
>

What confuses me is that everything seems to be where it needs to be: the conda and python binaries, the environment.

I checked the python directory:

ls /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/

and this is the result:

anndata                           loompy                        pip                              scipy
anndata-0.6.22.post1.dist-info    loompy-2.0.16-py3.7.egg-info  pip-20.2.2-py3.8.egg-info        scipy-1.5.2.dist-info
certifi                           natsort                       pkg_resources                    setuptools
certifi-2020.6.20-py3.6.egg-info  natsort-7.0.1.dist-info       __pycache__                      setuptools-49.6.0.post20200814-py3.6.egg-info
dateutil                          numpy                         pyparsing-2.4.7.dist-info        six-1.15.0.dist-info
_distutils_hack                   numpy-1.19.1.dist-info        pyparsing.py                     six.py
distutils-precedence.pth          packaging                     python_dateutil-2.8.1.dist-info  typing-3.5.2.2-py3.6.egg-info
easy_install.py                   packaging-20.4.dist-info      pytz                             typing.py
h5py                              pandas                        pytz-2020.1.dist-info            wheel
h5py-2.10.0.dist-info             pandas-1.1.1.dist-info        README.txt                       wheel-0.35.1-py3.6.egg-info

Now, the only thing I see different is the following:

py_config seems to return a "python 3.8.5" version, while conda_install() installs the package for 3.6.11.

So I tried to install python first on r-reticulate, in order to have it updated to the latest version, but then loompy downgrades it and I am back to square zero...

The fun thing is that with a combination of lines of code I was able to get a py_config with 3.6.11, but then again loompy is not available. If I try to install it, guess what? It upgrades python to 3.8.5.

In other words, I seem not to be ever able to use loompy that is installed in the directory:

/local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/

I am going crazy.

kevinushey commented 4 years ago

Here's my guess as to what's happening based on the shared code (with my comments above)

library(reticulate)

Sys.setenv(PYTHONHOME = "")
Sys.setenv(PYTHONPATH = "")

# reticulate::py_config()
reticulate::use_condaenv('r-reticulate', required = TRUE)
reticulate::conda_list()
reticulate::conda_binary()

# calling py_module_available here actually forces reticulate
# to initialize Python, so after this point reticulate has bound to
# whatever version of Python is available in the r-reticulate condaenv
reticulate::conda_install(packages = "numpy")
reticulate::py_module_available('numpy') 

# here, attempting to install loompy also seems to force a downgrade of Python
# note that this is happening _after_ reticulate has already initialized Python!
# so after this runs, reticulate has loaded Python 3.8, but now Python 3.6 is
# going to be installed in the requested environment
reticulate::conda_install(packages = "loompy", channel = 'bioconda')

# Python 3.8 is still used by reticulate, but Python 3.6 was just installed
# and a version of loompy for Python 3.6 was installed instead -- so
# it's not actually visible to reticulate right now
reticulate::py_module_available('loompy')
loompy <- reticulate::import('loompy')

reticulate::py_config()
reticulate::conda_list()
reticulate::conda_binary()

I think a big part of the problem here is that either reticulate is inadvertently forcing a different version of Python to be installed when loompy is installed, or loompy depends on a specific version of Python and that's forcing a downgrade.

micdonato commented 4 years ago

Here's my guess as to what's happening based on the shared code (with my comments above)

...

I think a big part of the problem here is that either reticulate is inadvertently forcing a different version of Python to be installed when loompy is installed, or loompy depends on a specific version of Python and that's forcing a downgrade.

I don't think loompy depends on a specific version. Look what happens when I don't force the 'r-reticulate' environment:


> library(reticulate)
> reticulate::conda_install(packages = "loompy", channel = 'bioconda')
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done

## Package Plan ##

  environment location: /local-scratch/users/myuser/miniconda3/envs/r-reticulate

  added / updated specs:
    - python

The following packages will be UPDATED:

  anndata            bioconda/noarch::anndata-0.6.22.post1~ --> conda-forge/linux-64::anndata-0.7.4-py38h32f6830_0
  loompy                       bioconda::loompy-2.0.16-py_0 --> conda-forge::loompy-2.0.17-py_0
  python                          3.6.11-h4d41432_2_cpython --> 3.8.5-h1103e12_7_cpython
  python_abi                                    3.6-1_cp36m --> 3.8-1_cp38
  typing                    bioconda::typing-3.5.2.2-py36_0 --> conda-forge::typing-3.7.4.3-py38h32f6830_0

The following packages will be DOWNGRADED:

  certifi                          2020.6.20-py36h9f0ad1d_0 --> 2020.6.20-py38h32f6830_0
  h5py                        2.10.0-nompi_py36hecadee3_104 --> 2.10.0-nompi_py38hfb01d0b_104
  numpy                               1.19.1-py36h3849536_2 --> 1.19.1-py38hbc27379_2
  pandas                               1.1.1-py36h831f99a_0 --> 1.1.1-py38h950e882_0
  scipy                                1.5.2-py36h3a855aa_0 --> 1.5.2-py38h8c5af15_0
  setuptools                          49.6.0-py36h9f0ad1d_0 --> 49.6.0-py38h32f6830_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done

## Package Plan ##

  environment location: /local-scratch/users/myuser/miniconda3/envs/r-reticulate

  added / updated specs:
    - loompy
    - python

The following packages will be SUPERSEDED by a higher-priority channel:

  anndata            conda-forge/linux-64::anndata-0.7.4-p~ --> bioconda/noarch::anndata-0.6.22.post1-py_0
  loompy                    conda-forge::loompy-2.0.17-py_0 --> bioconda::loompy-2.0.16-py_0
  typing             conda-forge::typing-3.7.4.3-py38h32f6~ --> bioconda::typing-3.5.2.2-py36_0

The following packages will be DOWNGRADED:

  certifi                          2020.6.20-py38h32f6830_0 --> 2020.6.20-py36h9f0ad1d_0
  h5py                        2.10.0-nompi_py38hfb01d0b_104 --> 2.10.0-nompi_py36hecadee3_104
  numpy                               1.19.1-py38hbc27379_2 --> 1.19.1-py36h3849536_2
  pandas                               1.1.1-py38h950e882_0 --> 1.1.1-py36h831f99a_0
  python                           3.8.5-h1103e12_7_cpython --> 3.6.11-h4d41432_2_cpython
  python_abi                                     3.8-1_cp38 --> 3.6-1_cp36m
  scipy                                1.5.2-py38h8c5af15_0 --> 1.5.2-py36h3a855aa_0
  setuptools                          49.6.0-py38h32f6830_0 --> 49.6.0-py36h9f0ad1d_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
> reticulate::py_module_available('loompy')
[1] FALSE
>

This is the weirdest part. First python is updated. Then it is downgraded.

kevinushey commented 4 years ago

Can you try your example once more with the development version of reticulate? You can install it with e.g.

remotes::install_github("rstudio/reticulate")

I just made a couple changes to conda_install() that I'm hopeful will solve this problem.

micdonato commented 4 years ago

Can you try your example once more with the development version of reticulate? You can install it with e.g.

remotes::install_github("rstudio/reticulate")

I just made a couple changes to conda_install() that I'm hopeful will solve this problem.

I just tried:

> remotes::install_github("rstudio/reticulate")
Skipping install of 'reticulate' from a github remote, the SHA1 (fd3a2610) has not changed since last install.
  Use `force = TRUE` to force installation
> library(reticulate)
> reticulate::conda_install(packages = "loompy", channel = 'bioconda')
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

> reticulate::py_module_available('loompy')
[1] FALSE
> reticulate::py_config()
python:         /usr/bin/python
libpython:      /usr/lib64/libpython2.7.so.1.0
pythonhome:     //usr://usr
version:        2.7.5 (default, Apr  2 2020, 13:16:51)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
numpy:           [NOT FOUND]
loompy:         [NOT FOUND]
> reticulate::conda_list()
          name                                                               python
1       python       /local-scratch/users/myuser/miniconda3/envs/python/bin/python
2 r-reticulate /local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python
> reticulate::conda_binary()
[1] "/local-scratch/users/myuser/miniconda3/bin/conda"
> reticulate::conda_python()
[1] "/local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python"

Why does it say that the module is there already?

Edit: I will try to force r-reticulate now.

Edit: Nothing, even forcing the environment and removing the package did not make it work.

library(reticulate)
>
> # Sys.setenv(PYTHONHOME = "")
> # Sys.setenv(PYTHONPATH = "")
>
> # reticulate::py_config()
> reticulate::use_condaenv('r-reticulate', required = TRUE)
> # reticulate::conda_list()
> # reticulate::conda_binary()
> #
> # reticulate::conda_install(packages = "numpy")
> # reticulate::py_module_available('numpy')
> #
> # reticulate::conda_install(packages = 'python')
>
> reticulate::conda_install(packages = "loompy", channel = 'bioconda')
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

> reticulate::py_module_available('loompy')
[1] FALSE
> reticulate::py_config()
python:         /local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python
libpython:      /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/libpython3.6m.so
pythonhome:     /local-scratch/users/myuser/miniconda3/envs/r-reticulate:/local-scratch/users/myuser/miniconda3/envs/r-reticulate
version:        3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42)  [GCC 7.5.0]
numpy:          /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/numpy
numpy_version:  1.19.1
loompy:         /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/loompy

NOTE: Python version was forced by use_python function
> reticulate::conda_list()
          name                                                               python
1       python       /local-scratch/users/myuser/miniconda3/envs/python/bin/python
2 r-reticulate /local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python
> reticulate::conda_binary()
[1] "/local-scratch/users/myuser/miniconda3/bin/conda"
> reticulate::conda_python()
[1] "/local-scratch/users/myuser/miniconda3/envs/r-reticulate/bin/python"
>
kevinushey commented 4 years ago

Dang, thanks. I'll try to take a closer look today.

kevinushey commented 4 years ago

Works for me:

library(reticulate)

# create new environment
conda_create("loompy-test")
#> [1] "/Users/kevinushey/opt/anaconda3/envs/loompy-test/bin/python"

# install loompy into that environment
conda_install(envname = "loompy-test",
              packages = "loompy",
              channel = "bioconda")

# activate that environment
use_condaenv("loompy-test", required = TRUE)

# check that we've activated it
py_config()
#> python:         /Users/kevinushey/opt/anaconda3/envs/loompy-test/bin/python
#> libpython:      /Users/kevinushey/opt/anaconda3/envs/loompy-test/lib/libpython3.8.dylib
#> pythonhome:     /Users/kevinushey/opt/anaconda3/envs/loompy-test:/Users/kevinushey/opt/anaconda3/envs/loompy-test
#> version:        3.8.5 | packaged by conda-forge | (default, Aug 29 2020, 01:18:42)  [Clang 10.0.1 ]
#> numpy:          /Users/kevinushey/opt/anaconda3/envs/loompy-test/lib/python3.8/site-packages/numpy
#> numpy_version:  1.19.1
#> 
#> NOTE: Python version was forced by use_python function

# is loompy available?
py_module_available("loompy")
#> [1] TRUE

Created on 2020-09-03 by the reprex package (v0.3.0)

micdonato commented 4 years ago

Ok then it must be some issue on my side... I am horrendously frustrated right now.

Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(reticulate)
> # create new environment
> conda_create("loompy-test")
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /local-scratch/users/myuser/miniconda3/envs/loompy-test

  added / updated specs:
    - python

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  ca-certificates    conda-forge/linux-64::ca-certificates-2020.6.20-hecda079_0
  certifi            conda-forge/linux-64::certifi-2020.6.20-py38h32f6830_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.34-hc38a660_9
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.3.0-h24d8f2e_16
  libgomp            conda-forge/linux-64::libgomp-9.3.0-h24d8f2e_16
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.3.0-hdf63c60_16
  ncurses            conda-forge/linux-64::ncurses-6.2-he1b5a44_1
  openssl            conda-forge/linux-64::openssl-1.1.1g-h516909a_1
  pip                conda-forge/noarch::pip-20.2.2-py_0
  python             conda-forge/linux-64::python-3.8.5-h1103e12_7_cpython
  python_abi         conda-forge/linux-64::python_abi-3.8-1_cp38
  readline           conda-forge/linux-64::readline-8.0-he28a2e2_2
  setuptools         conda-forge/linux-64::setuptools-49.6.0-py38h32f6830_0
  sqlite             conda-forge/linux-64::sqlite-3.33.0-h4cf870e_0
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0
  wheel              conda-forge/noarch::wheel-0.35.1-pyh9f0ad1d_0
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1009

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate loompy-test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

[1] "/local-scratch/users/myuser/miniconda3/envs/loompy-test/bin/python"
> # install loompy into that environment
> conda_install(envname = "loompy-test",
+               packages = "loompy",
+               channel = "bioconda")
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... done

## Package Plan ##

  environment location: /local-scratch/users/myuser/miniconda3/envs/loompy-test

  added / updated specs:
    - loompy
    - python

The following NEW packages will be INSTALLED:

  h5py               conda-forge/linux-64::h5py-2.10.0-nompi_py36hecadee3_104
  hdf5               conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_101
  libblas            conda-forge/linux-64::libblas-3.8.0-17_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-17_openblas
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-7.5.0-hdf63c60_16
  liblapack          conda-forge/linux-64::liblapack-3.8.0-17_openblas
  libopenblas        conda-forge/linux-64::libopenblas-0.3.10-pthreads_hb3c22a3_4
  loompy             bioconda/noarch::loompy-2.0.16-py_0
  numpy              conda-forge/linux-64::numpy-1.19.1-py36h3849536_2
  pandas             conda-forge/linux-64::pandas-1.1.1-py36h831f99a_0
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.1-py_0
  pytz               conda-forge/noarch::pytz-2020.1-pyh9f0ad1d_0
  scipy              conda-forge/linux-64::scipy-1.5.2-py36h3a855aa_0
  six                conda-forge/noarch::six-1.15.0-pyh9f0ad1d_0
  typing             bioconda/linux-64::typing-3.5.2.2-py36_0

The following packages will be DOWNGRADED:

  certifi                          2020.6.20-py38h32f6830_0 --> 2020.6.20-py36h9f0ad1d_0
  python                           3.8.5-h1103e12_7_cpython --> 3.6.11-h4d41432_2_cpython
  python_abi                                     3.8-1_cp38 --> 3.6-1_cp36m
  setuptools                          49.6.0-py38h32f6830_0 --> 49.6.0-py36h9f0ad1d_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
> # activate that environment
> use_condaenv("loompy-test", required = TRUE)
> # is loompy available?
> py_module_available("loompy")
[1] FALSE
>
kevinushey commented 4 years ago

Are you sure you're using the development version of reticulate? Did you restart R after installing the new package?

micdonato commented 4 years ago

Are you sure you're using the development version of reticulate? Did you restart R after installing the new package?

I restarted the session (Terminate R from RStudio). I will try to uninstall the package and re-install.

micdonato commented 4 years ago

Ok I call this progress:

> loompy <- reticulate::import('loompy')
Error in py_module_import(module, convert = convert) : 
  ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/scipy/_lib/_uarray/_uarray.cpython-36m-x86_64-linux-gnu.so)

Detailed traceback: 
  File "/local-scratch/users/myuser/RLibs/3.6/reticulate/python/rpytools/loader.py", line 24, in _import_hook
    level=level
  File "/local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/loompy/__init__.py", line 2, in <module>
    from .normalize import normalize_attr_values, materialize_attr_values
  File "/local-scratch/users/myuser/RLibs/3.6/reticulate/python/rpytools/loader.py", line 24, in _import_hook
    level=level
  File "/local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/loompy/normalize.py", line 2, in <module>
    import scipy.sparse as sparse
  File "/local-scratch/users/myuser/RLibs/3.6/reticulate/python/rpytools/loader.py", line 24, in _import

Maybe there is something weird with my system?

hanhyebin commented 3 years ago

loompy <- reticulate::import('loompy') Error in py_module_import(module, convert = convert) : ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/scipy/_lib/_uarray/_uarray.cpython-36m-x86_64-linux-gnu.so)

@micdonato were you able to figure it out? I have a similar error message: Error in py_module_import(module, convert = convert) : SyntaxError: invalid syntax (files.py, line 191)

micdonato commented 3 years ago

Nope, never was. I abandoned reticulate and rstudio and started using python packages directly from python.

It was a good run until it lasted.

Mic

On Sep 24, 2021, at 12:05, Hyebin Han @.***> wrote:

 loompy <- reticulate::import('loompy') Error in py_module_import(module, convert = convert) : ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /local-scratch/users/myuser/miniconda3/envs/r-reticulate/lib/python3.6/site-packages/scipy/_lib/_uarray/_uarray.cpython-36m-x86_64-linux-gnu.so)

@micdonato were you able to figure it out? I have a similar error message: Error in py_module_import(module, convert = convert) : SyntaxError: invalid syntax (files.py, line 191)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

hanhyebin commented 3 years ago

😭😭

micdonato commented 3 years ago

I want to clarify a couple of things though.

The system I was trying to use reticulate on was our lab server. I suspect the issue was on the server side but we never figured out what it was. I was using RStudio Server Pro, and I believe I had no issues when I tried with the Desktop version on my work laptop.

What was weird is that not even my sysadmin was able to figure out what happened.

hanhyebin commented 3 years ago

Interesting... I am having the opposite issue where I have issues with my desktop (I think it might be related to CPU) but not with our HPCC.

Regardless, @micdonato , thank you for your response and input! Best of luck 🥲🤞🏼

hasanmoudud commented 2 years ago

I want to clarify a couple of things though.

The system I was trying to use reticulate on was our lab server. I suspect the issue was on the server side but we never figured out what it was. I was using RStudio Server Pro, and I believe I had no issues when I tried with the Desktop version on my work laptop.

What was weird is that not even my sysadmin was able to figure out what happened.

I am having same problem when using reticulate on HPC server.

t-kalinowski commented 2 years ago

This type of issue can also be due to reticulate selecting different conda installations when initializing and when installing. In a recent PR, system commands installing into a conda environment are now echoed, so it should be easier to debug and diagnose if this is the cause. Please try with the dev version of reticulate and see if that helps:

remotes::install_github("rstudio/reticulate")
hasanmoudud commented 2 years ago

This type of issue can also be due to reticulate selecting different conda installations when initializing and when installing. In a recent PR, system commands installing into a conda environment are now echoed, so it should be easier to debug and diagnose if this is the cause. Please try with the dev version of reticulate and see if that helps:

remotes::install_github("rstudio/reticulate")

Not working yet.

remotes::install_github("rstudio/reticulate") Skipping install of 'reticulate' from a github remote, the SHA1 (214b90af) has not changed since last install. Use force = TRUE to force installation library(reticulate) conda_list() name 1 r-reticulate 2 base 3 greta-env python 1 /.local/share/r-miniconda/envs/r-reticulate/bin/python 2 /.local/share/r-miniconda/bin/python 3 /.local/share/r-miniconda/envs/greta-env/bin/python py_config() python: /.local/share/r-miniconda/envs/r-reticulate/bin/python libpython: /.local/share/r-miniconda/envs/r-reticulate/lib/libpython3.7m.so pythonhome: /.local/share/r-miniconda/envs/r-reticulate:/.local/share/r-miniconda/envs/r-reticulate version: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:21) [GCC 9.4.0] numpy: [NOT FOUND]

hasanmoudud commented 2 years ago

This type of issue can also be due to reticulate selecting different conda installations when initializing and when installing. In a recent PR, system commands installing into a conda environment are now echoed, so it should be easier to debug and diagnose if this is the cause. Please try with the dev version of reticulate and see if that helps:

remotes::install_github("rstudio/reticulate")

Not working yet.

remotes::install_github("rstudio/reticulate") Skipping install of 'reticulate' from a github remote, the SHA1 (214b90a) has not changed since last install. Use force = TRUE to force installation library(reticulate) conda_list() name 1 r-reticulate 2 base 3 greta-env python 1 /.local/share/r-miniconda/envs/r-reticulate/bin/python 2 /.local/share/r-miniconda/bin/python 3 /.local/share/r-miniconda/envs/greta-env/bin/python py_config() python: /.local/share/r-miniconda/envs/r-reticulate/bin/python libpython: /.local/share/r-miniconda/envs/r-reticulate/lib/libpython3.7m.so pythonhome: /.local/share/r-miniconda/envs/r-reticulate:/.local/share/r-miniconda/envs/r-reticulate version: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:21) [GCC 9.4.0] numpy: [NOT FOUND]

Problem solved:

module load R module unload GDAL/3.3.2-foss-2021b module unload pybind11/2.7.1-GCCcore-11.2.0 module unload SciPy-bundle/2021.10-foss-2021b

Yunuuuu commented 4 months ago

The same problems

Yunuuuu commented 4 months ago

reticulate can only found the python pakcage installed by system python binary with pip3, it cannot use conda environment.

t-kalinowski commented 4 months ago

@Yunuuuu, can you please open a new issue with instructions for how to reproduce the error you're seeing?

Yunuuuu commented 4 months ago

I thought my problem is the same with current issue and my test code is the same with this issue (I cannot use reprex with reticulate, I don't know why), here is my code examples:

First try:

options(reticulate.conda_binary = "/usr/local/bin/micromamba")
reticulate::conda_create("scanpy", "scanpy")
reticulate::import("scanpy")

Here is the output:

[R]> options(reticulate.conda_binary = "/usr/local/bin/micromamba")

[R]> reticulate::conda_create("scanpy", "scanpy")
+ /usr/local/bin/micromamba create --yes --name scanpy 'python=3.10' scanpy --quiet -c conda-forge
[1] "/home/pengyun/micromamba/envs/scanpy/bin/python"

[R]> reticulate::import("scanpy")
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'scanpy'
Run `reticulate::py_last_error()` for details.

[R]>

Second try

options(reticulate.conda_binary = "/usr/local/bin/micromamba")
reticulate::conda_create("scanpy", "scanpy")
# The requested version of Python cannot be used, as another version of
# Python ('/usr/bin/python3') has already been initialized.
Sys.setenv(RETICULATE_PYTHON = reticulate::conda_python("scanpy"))
reticulate::use_condaenv("scanpy")
reticulate::import("scanpy")

here is the output 


[R]> options(reticulate.conda_binary = "/usr/local/bin/micromamba")

[R]> reticulate::conda_create("scanpy", "scanpy")
+ /usr/local/bin/micromamba create --yes --name scanpy 'python=3.10' scanpy --quiet -c conda-forge
[1] "/home/pengyun/micromamba/envs/scanpy/bin/python"

[R]> # The requested version of Python cannot be used, as another version of
         # Python ('/usr/bin/python3') has already been initialized.
         Sys.setenv(RETICULATE_PYTHON = reticulate::conda_python("scanpy"))

[R]> reticulate::use_condaenv("scanpy")

[R]> reticulate::import("scanpy")
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'scanpy'
Run `reticulate::py_last_error()` for details.
[R]> sessionInfo()
R version 4.2.3 (2023-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=zh_CN.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=zh_CN.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] progressr_0.14.0       Rcpp_1.0.12            lattice_0.20-45       
 [4] png_0.1-8              fansi_1.0.6            digest_0.6.35         
 [7] utf8_1.2.4             withr_3.0.0            grid_4.2.3            
[10] lifecycle_1.0.4        jsonlite_1.8.8         pillar_1.9.0          
[13] rlang_1.1.3            cli_3.6.2              Matrix_1.5-1          
[16] vctrs_0.6.5            reticulate_1.37.0.9000 tools_4.2.3           
[19] glue_1.7.0             compiler_4.2.3
Yunuuuu commented 4 months ago

But I can fix this by specifying the environment in import:

reticulate::import("scanpy", delay_load = list(environment = "scanpy"))
t-kalinowski commented 4 months ago

The "Second try" approach is the one I would expect to work, except it seems that reticulate had already initialized another Python when you ran it. Can you please try it in a fresh R session?

Yunuuuu commented 4 months ago

It's the same:

[R]> options(reticulate.conda_binary = "/usr/local/bin/micromamba")

[R]> reticulate::conda_create("scanpy", "scanpy")
+ /usr/local/bin/micromamba create --yes --name scanpy 'python=3.10' scanpy --quiet -c conda-forge
[1] "/home/pengyun/micromamba/envs/scanpy/bin/python"

[R]> # The requested version of Python cannot be used, as another version of
     # Python ('/usr/bin/python3') has already been initialized.
     Sys.setenv(RETICULATE_PYTHON = reticulate::conda_python("scanpy"))

[R]> reticulate::use_condaenv("scanpy")

[R]> reticulate::import("scanpy")
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'scanpy'
Run `reticulate::py_last_error()` for details.
t-kalinowski commented 4 months ago 
[R]> # The requested version of Python cannot be used, as another version of
     # Python ('/usr/bin/python3') has already been initialized.

The error message is very useful here. It is telling you that something has already initiailzed Python. You must figure out why, and disable it. Most likely it is something in .Rprofile. It might be due to the RStudio IDE settings. py_config() will report what forced Python to initialize.