Allow builtin interpreter discovery to find specific Python versions given a general spec

[flying-sheep]

Thanks for contributing, make sure you address all the checklists (for details on how see development documentation)

• [x] ran the linter to address style issues (tox -e fix)
• [x] wrote descriptive pull request text
• [x] ensured there are test(s) validating the fix
• [x] added news fragment in docs/changelog folder
• [x] updated/extended the documentation

Fixes https://github.com/pypa/virtualenv/issues/2708, fixes https://github.com/pypa/hatch/issues/1395

This PR fixes the discovery in virtualenv.discovery.builtin.

It works by removing the method PythonSpec.generate_names and replacing it with generate_re:

A static name list can’t represent all Python executables that are compatible with a given spec, i.e. PythonSpec.from_string_spec('3') should match a executable called /usr/bin/python3.12. I therefore inverted the way candidates are found. Instead of generating all possible candidates, we simply loop and use a regex to find candidates.

Pro:

• Fixes the bug
• Instead of just trying all-lowercase and all-uppercase variants of candidates, we’re now truly case-insensitive

Con:

• Might be slower if a lot of executables hang out on PATH. Since Python’s glob just traverses directories in Python code as well, it wouldn‘t help here.

How to review

The main commits are

• f29be4a71d09afef4aece4adef89a2e47dd4086f: Fixes the tox configuration so it can cope with spaces. Allows me to run tests on my system. Humanity needs to stop interpolating strings into shell scripts.
• 6f9f270ebfa491cdf5d9c3a3bd97c9647d7a8a74: Adds typing to the file I’m going to edit so I can make sense of what’s going on
• 16a6e1d5a0ccf1aaf9f44400f4f8bd0eb0ced9c6: Switch that file to pathlib, since you seem to be doing that elsewhere and it makes things more readable
• d9b52ada0f67e8c85ce58bd6f46c5aa37e1201bd: The actual changes, described above

[flying-sheep]

On it! It’s a draft so you can look at it if you want while I check those last boxes!

[flying-sheep]

Now I’m done!

Seems like this change snuck into one of the tests:

-    pyvenv_cfg = Path(sys.executable).parents[1] / "pyvenv.cfg"
-    if pyvenv_cfg.exists():
-        (target / pyvenv_cfg.name).write_bytes(pyvenv_cfg.read_bytes())

that fragment doesn’t seem to be necessary to run the test locally?

[gaborbernat]

Seems you've upset the CI 😅

[flying-sheep]

I guess I need to try this on macOS. No idea what to do if I can’t placate windows though …

Seems like it works in practice though!

1. create project requiring Python 3.12 on a system where /usr/bin/python is 3.11:

   $ cd some-project
   $ rg requires-python pyproject.toml 
   4:requires-python = ">=3.12"

2. try finding an interpreter with hatch:

   $ pipx install hatch
   $ hatch env create
   Environment `default` is incompatible: no compatible Python distribution available

3. try my patch (no clue why virtualenv doesn’s:

   $ ~/.local/pipx/venvs/hatch/bin/python -m pip install 'virtualenv@ git+https://github.com/flying-sheep/virtualenv.git@discover-more'
   […]
   Successfully installed virtualenv-20.25.4.dev11+g66d9408
   $ hatch env create

   no output means success!

[gaborbernat]

Huh, tests run fine locally … let’s see.

I cannot merge the pull request until the CI is green so it doesn't really matter that "it works on my machine" 😅. Sorry but the CI gods are absolute. 😭

[flying-sheep]

Haha I’m a believer, don’t worry. It’s great that you test on windows and macOS too, as clearly with finnicky path stuff like this, that’s an issue. I’ll see what I can do.

[flying-sheep]

Ah, the regex always needs to be case insensitive to get the intended effect. I tested locally on macOS and Linux now, so I hope windows doesn’t throw a wrench.

[flying-sheep]

OK, as feared everything passes except for on windows. I can only speculate that the pyvenv.cfg stuff was load bearing on windows?

If that’s not it, someone with a windows machine needs to help.

/edit: e.g. @ofek, who just offered

[flying-sheep]

Looks good! I don’t think this timeout is caused by my code

[flying-sheep]

OK, pypy on windows fails with

125251 get interpreter info via cmd: 'C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_discovery_via_path_specif0\Scripts\somethingVeryCryptic3.10.exe' 'D:\a\virtualenv\virtualenv\.tox\pypy310\lib\site-packages\virtualenv\discovery\py_info.py' 2o7zKhwQ7RmUZlP1L7qYsRzWDWJd2Uc0 266eD4ebldJtJex0LCwbXGC040ZIuScj [DEBUG cached_py_info:112]
125280 failed to query C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_discovery_via_path_specif0\Scripts\somethingVeryCryptic3.10.exe with code 3221225781 [INFO cached_py_info:34]

I have absolutely no idea what “125280 failed to query …exe with code 3221225781” means, so I don’t think I can proceed here.