poetry env use X.Y fails on Windows

[absassi]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [ ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10

• Poetry version: 1.0.5

• Link of a Gist with the contents of your pyproject.toml file: not needed

Issue

The shortcut command poetry env use X.Y to select the environment for Python X.Y, as documented in Managing environments, is broken in Windows. One is required instead to find, copy and paste the full path to Python executable.

Note that the documentation even says "especially Windows where pyenv is not available".

>poetry env use 3.7
'python3.7' is not recognized as an internal or external command,
operable program or batch file.

The reason is quite simple: CPython installation in Windows doesn't provide pythonX.Y.exe, so even if you have all of them in PATH, Poetry can only find the first one. They do, however, provide registry keys with installation locations, which Poetry can use to resolve X.Y to the Python interpreter path.

This piece of code should find the path to Python, given the version number as a X.Y string:

import os
try:
    import winreg
except ImportError:
    import _winreg as winreg

def find_python_interpreter(version):
    for hkey, prefix in (
        (winreg.HKEY_CURRENT_USER, "Software"),
        (winreg.HKEY_CURRENT_USER, "Software\\WOW6432Node"),
        (winreg.HKEY_LOCAL_MACHINE, "Software"),
        (winreg.HKEY_LOCAL_MACHINE, "Software\\WOW6432Node"),
    ):
        try:
            value = winreg.QueryValue(hkey, "{}\\Python\\PythonCore\\{}\\InstallPath".format(prefix, version))
        except FileNotFoundError:
            continue
        python = os.path.join(value, "python.exe")
        if os.path.exists(python):
            return python
    return None

I have adjusted the code above to make it compatible with Python 2, but I have not tested afterwards, so please forgive any mistakes.

[konserw]

I've also been hit by this bug. Can this proposed solution be merged?

[shelper]

same here, it has been over a year, seems still not been fixed...

[radoering]

Not sure if we have to implement PEP 514 compliant searching by ourselves. It may be more sustainable to use an existing lib (e.g. pythonfinder).

[bruchar1]

Duplicates #3520.

[dimbleby]

4615 was closed on the grounds that it was a duplicate. #4615 also claimed that #4682 would be a fix - and that fix has since happened (at #5007).

So if that's all true then this can be closed.

(I don't use poetry on windows myself, so I didn't test)

[radoering]

4615 was closed on the grounds that it was a duplicate.

Probably wrong. This issue is still valid. One could debate whether it is a bug or a missing feature (+ documentation bug).

[timonviola]

In my opinion this is a bug and it's still present on poetry 1.4.2.

The introduction page says "It is multi-platform and the goal is to make it work equally well on Linux, macOS and Windows." It should be possible for Windows users to use the same commands.

[devmcp]

This still does not work on Windows on poetry 1.5.1

[Guillaume-Fgt]

Same here, poetry 1.5.1, all python versions in Path, Windows10:

C:\Users\guill>py --list
 -V:3.11 *        Python 3.11 (64-bit)
 -V:3.10          Python 3.10 (64-bit)
 -V:3.9           Python 3.9 (64-bit)
 -V:3.8           Python 3.8 (64-bit)

poetry env use 3.9
Could not find the python executable python3.9

poetry env use py -3.9
The option "3" does not exist

poetry env use python3.9
Could not find the python executable python3.9

but this one is working:

poetry env use C:\Users\guill\AppData\Local\Programs\Python\Python39\python.exe
Creating virtualenv vectorbt-backtesting in C:\Users\guill\Documents\Python\vectorbt_backtesting\.venv

[interceptor]

I have exactly the same issue, its quite annoying to be honest.

[mandar-karhade]

same issue here. Any progress on this one? currently on poetry 1.6.0 (Windows 11)

[AllanHOlesenBW]

I don't understand why poetry env use <version> doesn't just use py --list-paths to find the path to the desired python version on a Windows machine.

This command...:

py --list-paths

... gives this output on my machine:

-V:3.12          C:\prog\Python\Python312\python.exe
-V:3.11          C:\Prog\Python\Python311\python.exe
-V:3.10          C:\prog\Python\Python310\python.exe
-V:3.9           C:\prog\Python\Python39\python.exe
-V:3.8           C:\prog\Python\Python38\python.exe

It appears that py is a rather underrated tool. Its official name is "Python Launcher for Windows", and it comes with every python version since 3.6 (well, every version for Windows computers, that is). It is compatible with older versions too, if just one of the newer versions is installed. I never use python anymore. I always use py followed by a version and a command, for example:

py -3.11 -m venv ./.venv

[EliahKagan]

It seems to me that for poetry env use to use py --list-paths on Windows would be a step forward. However, it is not always present, since it is not included with the builds of Python provided by the Microsoft Store packages of Python. (I think it can also always be deselected explicitly when installing Python on Windows in the usual way, but this is less significant because people probably don't deselect it and, if they do, are likely aware that they have done so.)

[TBBle]

It'd be interesting to know if the MS Store Python install is PEP 514-compliant.

py as of Python 3.11 is a PEP 514 implementation, so anything PEP 514-compliant should show up in its --list output. I think integrating PEP 514 in some way (direct implementation or via another package) would be more robust (and easier) than trying to wrap the py launcher's output.

And that also means we don't get bitten by systems that don't have py installed for various reasons.

[AllanHOlesenBW]

It'd be interesting to know if the MS Store Python install is PEP 514-compliant.

I can't answer that. I don't have access to Microsoft Store, so I can't test it.

Edit: Wrong conclusion below. PEP 514 search would work right now, also for my older Python installations. The problem described below is caused by pythonfinder not supporting PEP514 though it claims to do so. Anyway, I just searched and found a package called pythonfinder, which claims to find PEP 514 compliant python installations on Windows PCs. On my PC it only finds Python 3.11 and 3.12 (which matches what you wrote), where py finds all my installed versions.

So 3 years from now, when everything before 3.11 is at end of life, PEP 514 may be a solution. But for now it will be of limited use.

[TBBle]

Which version of py are you using?

My understanding of PEP 514 was that the registry keys used by py were effectively grandfathered in, so a PEP 514-consuming tool (including 3.11-or-later-py) should see everything a pre-3.11 py sees and more. Specifically, PEP 514 did not change anything in the registry setup for officia Python releases, merely standardised the existing format and extended it for other vendors.

So your results suggest either I misunderstood something in PEP 514, or there's a bug in Pythonfinder. The Pythonfinder docs use Python 3.7 as an example, so they think they support it.

[AllanHOlesenBW]

Which version of py are you using?

I think I am using the version, which came with Python 3.12.0. But I couldn't find a way to make it reveal its version, and it is installed in a separate directory.

I just checked the registration database, and all my installed Python versions from 3.8 to 3.12 have the PEP 514 entries here: Computer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\

But I only have 3.11 and 3.12 in my path. So I wonder if Pythonfinder only uses the path, though it claims to use the registration database.

[TBBle]

I just had a quick look, and given I'm in a venv on a machine with only older Python installed, and no Python in my PATH...

> py --list
Installed Pythons found by C:\WINDOWS\py.exe Launcher for Windows
 (venv) *
 -3.10-64
 -3.9-64

I noticed that Finder.find_all_python_versions() returns an empty list, Finder().find_python_version(3, 9) returns nothing, and Finder().find_python_version(3, 10) and Finder().find_python_version() return the venv's Python.

So if you're testing from inside a venv, that might override its use of PEP 514, but not searching the PATH?

(Edit: That's probably "the running Python" being returned, rather than the venv's specifically... it makes sense that it'd always be detected when applicable, not sure if that can be disabled.)

But seems like a pythonfinder-side issue, anyway. (Probably important since pipx-based Poetry usage would be in a venv, but needs to find other versions... pythonfinder might need to expose this as an option; the existing options on the Finder constructor didn't seem to make a difference.)

Erk... Poking around, it looks like they dropped PEP 514 support in https://github.com/sarugaku/pythonfinder/pull/135... I'm not sure if that was intentional, it looks like their test suite might have been relying on Python being in the path (or they're only testing with pyenv, not system-installed Python) and they didn't notice that they'd dropped PEP 514 support, since they didn't remove it from the docs or mention it in the PR.

And sure enough, pip install "pythonfinder<2.0.0" and now Finder().find_all_python_versions() finds both my PEP-514 Python. And not the venv's... Finder(system=True).find_all_python_versions() doesn't find the venv's version either, maybe that was another 2.0.0 change?

Anyway, 100% a bug on pythonfinder's side. I don't have time to lodge a bug report right now (it's 3am here...) so if you get a chance to do so, that'd be great.

Edit: I also just noticed there's a pyfinder binary, but it fails to run as it appears to depend on click, which wasn't pulled in by pip... Looks like a bug in their project (or a limitation of their build tool?) that they have click corralled as an extra, but the pyfinder binary is not so-limited. Now I know to do that, testing is easier. ^_^

> pip install --quiet --upgrade "pythonfinder[cli]<2" && pyfinder --findall
Found python at the following locations:
3.10: 3.10.11 (64bit) @ C:\Program Files\Python310\python.exe
3.9: 3.9.10 (64bit) @ C:\Program Files\Python39\python.exe
> pip install --quiet --upgrade "pythonfinder[cli]" && pyfinder --findall
ERROR: No valid python versions found! Check your path and try again.
Please provide a command

Please provide a command is probably also a bug (missing elif?), it appears in the case that no Python versions are found.

I also guess it doesn't see the running binary because it's not called python.exe but pyfinder.exe, and ignore_unsuported in pyfinder.exe defaults to True; with no way to turn it off, as the flag option is missing the "disable" variant.

Late edit: Lodged https://github.com/sarugaku/pythonfinder/issues/158 for PEP-514 support being removed, and https://github.com/sarugaku/pythonfinder/issues/159 for the CLI issues I found while investigating that.

---

Alternative idea: Just directly use the module that pythonfinder used to use (https://github.com/zooba/pep514tools) to implement poetry env use X.Y? Basically avoids introducing any changed behaviours for non-Linux, as it simply replaces/enhances 'python'+ver with pep514tools.find(ver).info.install_path.executable_path or similar on Windows only. (Although that package has never had a release and no commits in 6 years. I don't know if that's "Gah", or maybe "Perfected"?)

Late edit: I lean towards either using pep514tools directly, or just open-coding a PEP-514 consumer implementation; "Find a Python binary by that name in the registry" should be fairly simple.

[LMS007]

can we please merge this ffs?

[Secrus]

can we please merge this ffs?

Merge what exactly? Also, please refrain from using foul language.

[allanolesen]

Merge what exactly? Also, please refrain from using foul language.

People express their surprise in different ways. I agree that the previous poster's language was not optimal. And I agree that there is nothing to merge.

With that said, some surprise is warranted. The status right now is:

1. The command poetry env use X.Y is broken on Windows if Python was installed with the official installer from python.org.
2. This was reported 4 years ago in this issue.
3. The original post in this issue contains the code needed for looking up the install path on a Windows machine according to PEP514.
4. The documentation still lists the poetry env use X.Y command without any warnings that it doesn't work in Windows.
5. This issue does not seem to be part of any roadmap. It looks entirely abandoned. There is even a post where dimbleby asked if he could close it.

I would create a pull request myself, but I haven't figured out from where in the poetry code I would need to call the function. The exception, which is shown to the user when X.Y is not found seems to originate from another part of the code than the one I would think was used by poetry env.

So right now I am contemplating creating a pull request for a documentation update telling the user that the command is broken.

[allanolesen]

Late edit: I lean towards either using pep514tools directly, or just open-coding a PEP-514 consumer implementation; "Find a Python binary by that name in the registry" should be fairly simple.

Well, it is actually so simple that the code is in the very first post in this issue. Though with a hardcoded vendor name, which may not be the best approach.

Anyway, I have also poked around. Here are my findings:

The installer from python.org...:

• WILL add the PEP514 keys to the registry
• will NOT install an executable named pythonX.Y.exe
• will NOT add the python install folder to the path unless the user marks a checkbox

Installing from Windows Store...:

• will NOT add the PEP514 keys to the registry
• WILL install an executable named pythonX.Y.exe
• NOT add the python install folder to the path, but the above mentioned .exe will be installed in a folder, which already is in the path: C:\Users\%username%\AppData\Local\Microsoft\WindowsApps\

I don't know if Microsoft Store will install the py launcher. It was already installed on the PC I tested on (running Windows 11 Home). But from what I have read, it probably will not be installed.

The command poetry env use X.Y will find the python versions installed by Microsoft Store, but not the ones installed by the official installer from python.org.

The command py --list-paths will find all versions. So it must be searching in more than one way.

[radoering]

So right now I am contemplating creating a pull request for a documentation update telling the user that the command is broken.

The command is not broken. Passing X.Y is just not supported on Windows. Passing a path to a python.exe works fine. In other words, a docs update is welcome but it will not be accepted if it says "the command is broken".

This issue does not seem to be part of any roadmap. It looks entirely abandoned.

A solution would be nice for sure but the issue has not been important enough for anyone, perhaps because there is a simple workaround: just pass the path to the python.exe.

Anyway, I have also poked around.

IIUC, Microsoft Store Python is not compliant with PEP 514. However, that does not matter because they create a pythonX.Y executable in a directory that is in the PATH.

Originally, I said that it might make more sense to use a library like pythonfinder than to implement PEP 514 for ourselves. However, it looks like they dropped PEP 514 support (cf sarugaku/pythonfinder#158). In the meantime, I think it might make more sense to just implement it in Poetry.

In case, you are interested in creating a PR, the relevant part in the code should be here:

https://github.com/python-poetry/poetry/blob/cab9c42a0a09fd7aa22886254258d37c8fb931d3/src/poetry/utils/env/env_manager.py#L177-L188

[Secrus]

In the meantime, I think it might make more sense to just implement it in Poetry.

There is a plan to adopt findpython, which provides proper windows support afaik, some work to make that possible is happening at #9050.

[allanolesen]

There is a plan to adopt findpython, which provides proper windows support afaik

Have you read the posts about findpython from TBBIe and myself in this thread?

The developers of findpython dropped PEP514 support. So unless you limit findpython to old versions, or PEP514 support is reinstated in findpython, that approach seems like a dead end.

[Secrus]

@allanolesen there is pythonfinder and findpython. Those are 2 separate libraries.

[allanolesen]

there is pythonfinder and findpython. Those are 2 separate libraries.

Oops. Sorry for the confusion.

[jer-tx]

this still seems to be broken in windows 11, any updates?