Should preserve symlink to specified python interpreter

[bulletmark]

This describes an issue where uv venv operates in an incompatible and less-optimal way compared to python -m venv.

$ uv --version
uv 0.1.6

I use pyenv to install multiple python versions etc. I don't use the pyenv shims and just specify the pyenv python executable when building my venv's. So I have symlinks to the latest minor versions, i.e:

$ cd ~/.pyenv/versions
$ ls -l
lrwxrwxrwx - mark mark 13 Jan 12:47 3.7 -> 3.7.17/
drwxr-xr-x - mark mark 13 Jun  2023 3.7.17/
lrwxrwxrwx - mark mark 13 Jan 12:47 3.8 -> 3.8.18/
drwxr-xr-x - mark mark 29 Aug  2023 3.8.18/
lrwxrwxrwx - mark mark 13 Jan 12:47 3.9 -> 3.9.18/
drwxr-xr-x - mark mark 29 Aug  2023 3.9.18/
lrwxrwxrwx - mark mark 13 Jan 12:47 3.10 -> 3.10.13/
drwxr-xr-x - mark mark 29 Aug  2023 3.10.13/
lrwxrwxrwx - mark mark 18 Feb 13:04 3.11 -> 3.11.8/
drwxr-xr-x - mark mark 18 Feb 13:04 3.11.8/
lrwxrwxrwx - mark mark 21 Feb 11:32 3.12 -> 3.12.2/
drwxr-xr-x - mark mark 21 Feb 11:18 3.12.1/
drwxr-xr-x - mark mark 18 Feb 12:53 3.12.2/

Now witness the difference in the created symlinks to the interpreter when making a venv using these links:

$ ~/.pyenv/versions/3.12/bin/python -m venv venv
$ ls -l venv/bin/python
lrwxrwxrwx - mark mark 21 Feb 12:36 venv/bin/python -> /home/mark/.pyenv/versions/3.12/bin/python*

$ uv venv -p ~/.pyenv/versions/3.12/bin/python uvenv
Using Python 3.12.2 interpreter at /home/mark/.pyenv/versions/3.12.2/bin/python3.12
Creating virtualenv at: uvenv
Activate with: source uvenv/bin/activate
$ ls -l uvenv/bin/python
lrwxrwxrwx - mark mark 21 Feb 12:42 uvenv/bin/python -> /home/mark/.pyenv/versions/3.12.2/bin/python3.12

I.e. python -m venv creates a 3.12 link as I specified but uv venv dereferences that 3.12 path and creates a final link to 3.12.2.

This is undesirable because it means when 3.12 gets it's next compatible maintenance release, e.g. 3.12.3 and the pyenv link is updated, then the python -m venv venv will automatically use that 3.12.3 update but the uv venv uvenv will continue to use the older release and requires the venv to be rebuilt against the new version.

So is there any reason uv venv needs to resolve that symlink? It seems better to do what python -m venv does and just use it directly.

[charliermarsh]

This was changed in https://github.com/astral-sh/uv/pull/966 to fix something else. Maybe @konstin knows if we actually need to resolve symlinks there, or if we just needed to convert to an absolute path or something.

[konstin]

We need to resolve symlinks for the caching, but i think we can use the resolved version for caching and the original version for the venv. I think we should special case venv-from-venv cases, otherwise the new venv will break when deleting the venv it was created from.

[charliermarsh]

This is related to a few other issues, but note that virtualenv (on main) also resolves symlinks like this.

[charliermarsh]

For the future: similar to https://github.com/astral-sh/uv/issues/1640. Need to decide if we want to preserve symlinks in these various cases.

[gschaffner]

$ ~/.pyenv/versions/3.12/bin/python -m venv venv
$ ls -l venv/bin/python
lrwxrwxrwx - mark mark 21 Feb 12:36 venv/bin/python -> /home/mark/.pyenv/versions/3.12/bin/python*

$ uv venv -p ~/.pyenv/versions/3.12/bin/python uvenv
Using Python 3.12.2 interpreter at /home/mark/.pyenv/versions/3.12.2/bin/python3.12
Creating virtualenv at: uvenv
Activate with: source uvenv/bin/activate
$ ls -l uvenv/bin/python
lrwxrwxrwx - mark mark 21 Feb 12:42 uvenv/bin/python -> /home/mark/.pyenv/versions/3.12.2/bin/python3.12

2327 is related—when creating the same uv venv but without passing --python, uv's report states that uv venv didn't resolve versions/3.12 to versions/3.12.2:

+ type python
python is /home/ganden/.pyenv/shims/python
+ python -c 'import sys; print(sys.executable)'
/home/ganden/.pyenv/versions/3.12/bin/python
+ readlink -f /home/ganden/.pyenv/versions/3.12/bin/python
/home/ganden/.pyenv/versions/3.12.2/bin/python3.12
+ : '^^^ the above resolution is due to the symlinks'
+ readlink /home/ganden/.pyenv/versions/3.12 /home/ganden/.pyenv/versions/3.12.2/bin/python
3.12.2
python3.12
+ :
+ uv venv
Using Python 3.12.2 interpreter at: /home/ganden/.pyenv/versions/3.12/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
+ : '^^^ uv reports using versions/3.12'
+ readlink .venv/bin/python
/home/ganden/.pyenv/versions/3.12.2/bin/python3.12
+ : '^^^ but actually uses versions/3.12.2'

[charliermarsh]

I'm not totally sure what we're supposed to do when creating a virtualenv from within a virtualenv.

[charliermarsh]

python -m venv seems to use the existing virtualenv as the base:

❯ cat .nested-venv/pyvenv.cfg
home = /Users/crmarsh/workspace/uv/.venv/bin
include-system-site-packages = false
version = 3.8.18

However, virtualenv does not seem to do this.

[CharString]

I this is how it's supposed to work, then what's the proposed workflow to upgrade, say, 3.9.18 to 3.9.19 and repair all my venvs that I created to use 3.9? freeze, rm, venv, sync?

[charliermarsh]

Yeah, that's technically the correct thing to do, because you could have packages in your environment that are compatible with the previous but not the updated patch version; or packages whose dependencies differ by patch version. Both of those things are allowed in Python / per the spec though are very rare in practice when talking about patch versions.

[charliermarsh]

(We may change our behavior here eventually to preserve the symlink; undecided right now.)

[CharString]

Hmmm, right. Then maybe the correct thing would be the addition of some venv rebuild command, the way pipx has the reinstall and reinstall-all.

[ncoghlan]

As far as creating virtualenvs from within virtualenvs goes, the python -m venv behaviour is questionable, since it won't work in the general case (if you want to layer virtual environments together in a way that actually works, you currently need to add a sitecustomize.py file to the upper layers to get everything resolving correctly at runtime, simply referencing a venv layout as your "base Python" environment may not work reliably because some of the files can end up in the wrong relative locations, especially when Python has been compiled as a shared library with a wrapper executable). (I haven't gone looking to see if there's an open bug report about this though)

Outside the specific scenario of being passed a virtual environment symlink, using a provided symlink as given rather than resolving it feels like it would be more correct, though.

[charliermarsh]

Interesting. Perhaps what we want then, is: resolve the symlink if it's a virtualenv, preserve it if not?