Open GreenGrassBlueOcean opened 2 years ago
I have forked the project and added .globals$py_config <- initialize_python(some variables)
to use_python
:
https://github.com/GreenGrassBlueOcean/reticulate/commit/b3f1abf9acf71213611347e6b633ef203c59f197
Now reticulate::py_config()
shows the correct conda environment.
But still the python module cannot be found.
> use_miniconda(condaenv = "r-eikon", required = T)
> reticulate::py_config()
python: C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon/python.exe
libpython: C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon/python38.dll
pythonhome: C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon
version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon/Lib/site-packages/numpy
numpy_version: 1.22.1
NOTE: Python version was forced by use_python function
> reticulate::py_module_available(module = "eikon")
[1] FALSE
However in the miniconda terminal the package can be spotted. This means that still not the correct conda environment is used.
(base) PS C:\Users\XXXX> conda activate r-eikon
(r-eikon) PS C:\Users\XXXX> conda list
# packages in environment at C:\Users\XXXX\AppData\Local\R-MINI~1\envs\r-eikon:
#
# Name Version Build Channel
anyio 3.5.0 pypi_0 pypi
appdirs 1.4.4 pypi_0 pypi
ca-certificates 2021.10.8 h5b45459_0 conda-forge
certifi 2021.10.8 pypi_0 pypi
chardet 3.0.4 pypi_0 pypi
charset-normalizer 2.0.11 pypi_0 pypi
datetime 4.3 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
deprecation 2.1.0 pypi_0 pypi
eikon 1.1.6.post2 pypi_0 pypi
eventemitter 0.2.0 pypi_0 pypi
h11 0.12.0 pypi_0 pypi
h2 3.2.0 pypi_0 pypi
hpack 3.0.0 pypi_0 pypi
httpcore 0.13.7 pypi_0 pypi
httpx 0.19.0 pypi_0 pypi
hyperframe 5.2.0 pypi_0 pypi
idna 2.10 pypi_0 pypi
nest-asyncio 1.3.3 pypi_0 pypi
numpy 1.22.1 pypi_0 pypi
openssl 3.0.0 h8ffe710_2 conda-forge
packaging 21.3 pypi_0 pypi
pandas 1.4.0 pypi_0 pypi
pip 22.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.7 pypi_0 pypi
python 3.8.12 h900ac77_2_cpython conda-forge
python-configuration 0.8.2 pypi_0 pypi
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.8 2_cp38 conda-forge
pytz 2021.3 pypi_0 pypi
refinitiv-dataplatform 1.0.0a13 pypi_0 pypi
requests 2.27.1 pypi_0 pypi
rfc3986 1.5.0 pypi_0 pypi
scipy 1.7.3 pypi_0 pypi
setuptools 60.5.0 py38haa244fe_0 conda-forge
six 1.16.0 pypi_0 pypi
sniffio 1.2.0 pypi_0 pypi
sqlite 3.37.0 h8ffe710_0 conda-forge
ucrt 10.0.20348.0 h57928b3_0 conda-forge
urllib3 1.26.8 pypi_0 pypi
validators 0.18.2 pypi_0 pypi
vc 14.2 hb210afc_6 conda-forge
vs2015_runtime 14.29.30037 h902a5da_6 conda-forge
watchdog 2.1.6 pypi_0 pypi
websocket-client 1.2.3 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
zope-interface 5.4.0 pypi_0 pypi
(r-eikon) PS C:\Users\
Hi @t-kalinowski,
please any response?
Hi, apologies for the taking so long to respond. I'm not quite sure I understand the issue.
Changing reticulate:::.globals$py_config
won't change which Python reticulate bound to.
When reticulate is first loaded, reticulate:::.globals$py_config
is NULL
, and then that value gets updated when Python is initialized. Python can only be initialized once per R session; support for dynamically binding to and releasing multiple Pythons in one R session is not supported, and probably won't ever be. Updating the value of reticulate:::.globals$py_config
once Python is initialized won't change the fact that the Python session is initialized already.
@t-kalinowski Thanks a lot for your reply, much appreciated!
The issue occurs if you want to use two conda environments the default environment r-reticulate
and a new conda environment r-eikon
only for the python packages required by an r-package.
In that case reticulate::use_miniconda(condaenv = "r-eikon", required = T)
will not switch to the new environments. This is proven by calling reticulate::py_module_available
afterwards. This will only return TRUE
if the python packages are listed in r-reticulate
and FALSE
when they are in the new environment r-eikon
and not also in r-reticulate
.
To workaround this you need to install all packages in the default environment r-reticulate
which you don't want as you want to isolate the specific python packages I want to use from other already installed packages in r-reticulate
environment. This is done in order to prevent dependency issues with already installed packages in r-reticulate
.
While reading your response I was starting to get the (maybe wrong) impression that making a custom conda environment for an r package becomes useless as this cannot be used.
Hope this clarifies it a bit. It is indeed a complex issue. If my explanation is still lacking clarity please let me know!
I think that when you are writing an R package that uses reticulate, you can aim for convenience by hooking into reticulate::configure_environment, and/or by calling use_condaenv("r-eikon", required = FALSE)
in .onLoad()
. However requiring that the package works only with one specific python installation feels overly restrictive.
In the tensorflow ecosystem we like to provide a install_<name>
function that users can call to install the Python modules into their Python installation of choice.
In all cases though, once reticulate's Python session is initialized and bound to one Python, it cannot then initialize another Python. If you absolutely need to use only one Python conda environment in your R package, then you would need to raise an error if "wrong" Python is already initialized.
Many thanks for your insightful reply!
I have written an install_eikon
function (similar to that of tensorflow) that creates a conda environment with only the required python packages. However I guess from reading your reply it is maybe better to just make a virtual environment (venv) inside the users' default conda environment and activate this virtual environment when loading the package.
Because I guess the chances are quite high that the default python environment is already activated when a user is loading my package.
The reason that I am advocating for using conda or venv is that I am very afraid of python package version dependency issues inside the user's default python session, this can possibly lead to all kind of unexpected behavior inside my r package which is very hard to predict and to check for in unit tests. In my eyes having a shielded python environment would be highly desirable.
I would like to end with the fact that reticulate is an amazing package which allows me to run many python packages inside r. Thanks a lot for this package.
Hi @kevinushey and @t-kalinowski,
I am working on an r-package and I want to use a specific miniconda python environment in the r package. I notice however that after I use
reticulate::use_miniconda(condaenv = "r-eikon", required = T)
the internal global package value.globals$required_python_version
is updated to the r-eikon conda environment but the value of.globals$py_config
is left at the old conda environment (r-reticulate).This leads to very weird behavior. E.g. when calling
reticulate::py_module_available
will return that already installed python modules are not available. However when opening miniconda and runningconda activate r-eikon
andconda list
these are shown as installed in the r-eikon environment.A possible solution for this would be to again initialize_python as currently it is only initialized once and when changing environments it is not initialized again. https://github.com/rstudio/reticulate/blob/da515709ea629f5a9258bf35a1e4632a03e642a2/R/package.R#L29-L56
So maybe in
use_python()
after the following line: https://github.com/rstudio/reticulate/blob/1523ef58a13ba987267e5f976813525bedd17366/R/use_python.R#L112When there is a difference between
.globals$required_python_version
and.globals$py_config
to add something analogue to this:
.globals$py_config <- initialize_python(some variables)
Reproducible bug: