Could not find python environment when installing package on windows using external pyenv installation

[cderv]

I am in a context of pyenv managed python, installed on its own and not with reticulate

I tried to install package for this installation from R with py_install() but got this strange error

> library(reticulate)
> py_discover_config()
python:         C:/Users/chris/.pyenv/pyenv-win/shims/python3.bat
libpython:      C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/python311.dll
pythonhome:     C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1
version:        3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/Lib/site-packages/numpy
numpy_version:  1.23.2

python versions found: 
 C:/Users/chris/.pyenv/pyenv-win/shims/python3.bat
 C:/Users/chris/.pyenv/pyenv-win/shims/python.bat
> py_install("plotly")
Error:
! could not find a Python environment for C:/
Run `rlang::last_error()` to see where the error occurred.
> rlang::last_error()
<error/rlang_error>
Error:
! could not find a Python environment for C:/
---
Backtrace:
 1. reticulate::py_install("plotly")
 2. reticulate:::python_info(python)
 3. reticulate:::stopf(...)
Run `rlang::last_trace()` to see the full context.

I believe this is because I am using pyenv installed from Scoop (😅 )

python <- tryCatch(reticulate::py_exe(), error = function(e) NULL)
python
#> [1] "C:/Users/chris/.pyenv/pyenv-win/shims/python3.bat"

and reticulate only checks for python.exe in specific folder and its parents https://github.com/rstudio/reticulate/blob/0c57dc91f98fca814cc3b781a782984e308a15fa/R/python-tools.R#L47-L53

If somehow reticulate could detect that pyenv is used, it could be leveraged to find the python executable to use (Scoop or not scoop) ?

normalizePath(system("pyenv which python", intern = TRUE))
#> [1] "C:\\Users\\chris\\.pyenv\\pyenv-win\\versions\\3.11.1\\python.exe"

Also, error message is not clear,

! could not find a Python environment for C:/

I did not understood at first why C:/ was there, now I see this is because it goes from py_exe() place to the root folder without finding python.exe.

Happy to help improve support for pyenv

[t-kalinowski]

Hi @cderv,

I tried to setup a Windows machine w/ pyenv-win, using scoop, but it appears that scoop doesn't work w/ pyenv-win presently (https://github.com/pyenv-win/pyenv-win/issues/449). Unfortunately, rolling back (as suggested here https://github.com/ScoopInstaller/Main/issues/4143#issuecomment-1331785583) doesn't work for a different reason (https://github.com/pyenv-win/pyenv-win/issues/383).

Do you think this is specific to scoop + pyenv, or that this is a more general pyenv issue? If you think you understand how to fix, I'd be happy to review and accept a PR. Alternatively, we can try and reproduce on a fresh windows image on AWS.

[cderv]

I believe this is because I am using pyenv installed from Scoop (😅 )

And it was a mistake to say that, and my example shows it was wrong.

You are right that scoop pyenv installer does work so I switch to install pyenv a more classique way. I was mistaken because pyenv-win also uses shims with .bat file which is the issue here.

Do you think this is specific to scoop + pyenv, or that this is a more general pyenv issue?

This means it is not specific to scoop then

If you think you understand how to fix, I'd be happy to review and accept a PR. Alternatively, we can try and reproduce on a fresh windows image on AWS.

I can probably come up with a fix yes. Especially if we are open to ue pyenv which python to get the active python version when pyenv is detected, instead of looking into folders.

BTW I can render document with python and run reticulate - I got the issue only when trying to install package. So somehow code path is a bit different according to the use case (running python code, or running pip install ?)

It won't be very soon as I need to prioritize other things but I can do such work later. You can assign me on this

[t-kalinowski]

Do you use the pyenv built python directly in projects? I've been mainly using pyenv built pythons as a base for constructing virtual environments.

How do you specify intent for reticulate which python you want it to bind to?

[cderv]

Do you use the pyenv built python directly in projects?

Yes I think so.

I am not using a virtualenv outside of specific project. Usually my main install is where I do my common python work.

So I installed python using pyenv and I have two versions for now and I have set using pyenv global my default one to 3.11.1

> pyenv versions
* 3.11.1 (set by C:\Users\chris\.pyenv\pyenv-win\version)
  3.9.3

So this is the python used by default each time I am running python - I expect it to be run in terminal, with Quarto or with rmarkdown. reticulate seems to see it too

> library(reticulate)
> py_discover_config()
python:         C:/Users/chris/.pyenv/pyenv-win/shims/python3.bat
libpython:      C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/python311.dll
pythonhome:     C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1
version:        3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/Lib/site-packages/numpy
numpy_version:  1.23.2

Should I use a virtualenv without question ? I create virtualenv only on specific project.

I guess I have the workflow as with renv - I use default main R library for all, except specific project where I use renv.

For example this is where my jupyter is

> pip show jupyter
Name: jupyter
Version: 1.0.0
Summary: Jupyter metapackage. Install all the Jupyter components in one go.
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.org
License: BSD
Location: C:\Users\chris\.pyenv\pyenv-win\versions\3.11.1\Lib\site-packages
Requires: ipykernel, ipywidgets, jupyter-console, nbconvert, notebook, qtconsole
Required-by:

as I use this main python environment with Quarto too.

Wrong workflow ?

[t-kalinowski]

Wrong workflow ?

No such thing :) I'm only trying to understand the usage pattern. Thank you for the explanation, that's very helpful.

[t-kalinowski]

@cderv Is this still an issue you're encountering?

[cderv]

Same issue if I am using exactly the same workflow getting me to this install

> py_discover_config()
python:         C:/Users/chris/.pyenv/pyenv-win/shims/python3.bat
libpython:      C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/python311.dll
pythonhome:     C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1
version:        3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/Lib/site-packages/numpy
numpy_version:  1.23.2

NOTE: Python version was forced by PATH

python versions found: 
 C:/Users/chris/.pyenv/pyenv-win/shims/python3.bat
 C:/Users/chris/.pyenv/pyenv-win/shims/python.bat

> py_install("plotly")
Error:
! could not find a Python environment for C:/
Run `rlang::last_trace()` to see where the error occurred.

Though I pass with virtualenv_install() it works and will correctly create the default virtual env before installing

> reticulate::virtualenv_install(packages = "plotly")
Using Python: C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/python.exe
Creating virtual environment "~/.virtualenvs/r-reticulate" ... 
+ "C:/Users/chris/.pyenv/pyenv-win/versions/3.11.1/python.exe" -m venv "C:/Users/chris/Documents/.virtualenvs/r-reticulate"
Done!
Installing packages: pip, wheel, setuptools
+ "C:/Users/chris/Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -m pip install --upgrade pip wheel setuptools
Requirement already satisfied: pip in c:\users\chris\docume~1\virtua~1\r-reti~1\lib\site-packages (22.3.1)
Collecting pip
  Using cached pip-23.3.2-py3-none-any.whl (2.1 MB)
Collecting wheel
  Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
Requirement already satisfied: setuptools in c:\users\chris\docume~1\virtua~1\r-reti~1\lib\site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-69.0.3-py3-none-any.whl (819 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-23.3.2 setuptools-69.0.3 wheel-0.42.0
WARNING: There was an error checking the latest version of pip.
Virtual environment '~/.virtualenvs/r-reticulate' successfully created.
Using virtual environment "~/.virtualenvs/r-reticulate" ...
+ "C:/Users/chris/Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -m pip install --upgrade --no-user plotly
Collecting plotly
  Using cached plotly-5.18.0-py3-none-any.whl.metadata (7.0 kB)
Collecting tenacity>=6.2.0 (from plotly)
  Using cached tenacity-8.2.3-py3-none-any.whl.metadata (1.0 kB)
Collecting packaging (from plotly)
  Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
Using cached plotly-5.18.0-py3-none-any.whl (15.6 MB)
Using cached tenacity-8.2.3-py3-none-any.whl (24 kB)
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Installing collected packages: tenacity, packaging, plotly
Successfully installed packaging-23.2 plotly-5.18.0 tenacity-8.2.3

And then using default virtualenv, works

> py_install("matplotlib")
Using virtual environment "C:/Users/chris/Documents/.virtualenvs/r-reticulate" ...
+ "C:/Users/chris/Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -m pip install --upgrade --no-user matplotlib
(...) 

So this is a matter of workflow - difference in how python exe is found in py_install() and virtualenv_install() maybe.

Anyhow, non blocking when you know how to setup.

[t-kalinowski]

I'm not set-up with a windows PC at the moment, so I can't easily debug/diagnose. What does rlang::last_trace() report? If you step through py_install(), do you know where the error is?

We can diagnose IRL next week easily too.

[cderv]

I can look into it again, though OP has information about the trace and problem.

Error is from python_info() which looks for python.exe moving up into parent until C:/ and so does not the the shims from pyenv.

It seems virtualenv_install() has another logic to find python version to use.