Virtual environments break on Homebrew upgrades due to using a `Cellar` link

cjolowicz commented 8 months ago

On Homebrew, virtual environments created by uv venv reference the Python installation under Cellar in their interpreter symlink and pyvenv.cfg, which has the full downstream version in its path. These virtual environments break when Homebrew upgrades the respective Python package to the next maintenance release. In recent versions of venv and virtualenv, this issue was resolved by using the stable link under $(brew --prefix)/opt/python@3.x/ instead. For example, on Python 3.11 macOS aarch64 this would be the interpreter in /opt/homebrew/opt/python@3.11/bin.

This shell session demonstrates the problem:

❯ uv venv
❯ readlink .venv/bin/python
/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin/python3.11
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin

When Homebrew upgrades its python@3.11 package and cleans up the old installation under Cellar, those references in the virtual environment start to dangle.

For comparison, here's what I get with venv from Homebrew's python@3.11 installation:

❯ python3.11 -m venv .venv
❯ readlink .venv/bin/python
python3.11
❯ readlink .venv/bin/python3.11
/opt/homebrew/opt/python@3.11/bin/python3.11
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.11/bin

And with virtualenv:

❯ virtualenv --version
virtualenv 20.25.0 from ~/.local/pipx/venvs/virtualenv/lib/python3.12/site-packages/virtualenv/__init__.py
❯ virtualenv .venv
❯ readlink .venv/bin/python
/opt/homebrew/opt/python@3.12/bin/python3.12
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.12/bin

Affected platforms: Linux and macOS with Homebrew Python

❯ uv --version
uv 0.1.4
❯ gcc -dumpmachine
arm64-apple-darwin23.2.0

charliermarsh commented 8 months ago

Do you by any chance have a link to the relevant PRs or issues in venv and/or virtualenv?

cjolowicz commented 8 months ago

I don't, but FWIW venv derives the home key from sys._base_executable, which is the stable path.

charliermarsh commented 8 months ago

Hmm yeah, we're using sys._base_executable as of an open PR, but even that gives me:

❯ python3.8
Python 3.8.18 (default, Aug 24 2023, 19:48:18)
[Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/opt/homebrew/bin/python3.8'

charliermarsh commented 8 months ago

The problem is that we're calling canonicalize on the sys.executable.

charliermarsh commented 8 months ago

@konstin -- We may want to revisit this change (https://github.com/astral-sh/uv/issues/965). I think we should only do this (canonicalize) if we're in a virtual environment?

charliermarsh commented 8 months ago

We may want something like...

If we're not in a venv, use sys.executable.
If we are in a venv, resolve symlinks once (not recursively) on sys.executable.

That's closer to virtualenv:

# if we're not in a virtual environment, this is already a system python, so return the original executable
# note we must choose the original and not the pure executable as shim scripts might throw us off
return self.original_executable

ofek commented 8 months ago

I don't know if it's quite relevant but they just fixed a bug by storing the resolved absolute path in the metadata file that is generated for virtual environments: https://github.com/pypa/virtualenv/issues/2682

Also, I know all of you are basically Rust experts, but in case I am reading the source correctly please never ever use canonicalize directly as it is literally (I'm not joking) broken on Windows and will cause all of us bugs and unexpected behavior. Please don't use it. Instead, it's common to use the dunce crate or the normpath crate when you don't want to resolve symlinks. See even Armin talking about it.

charliermarsh commented 8 months ago

Haha. We do use fs::canonicalize but we then strip UNC paths everywhere. (I guess we could use dunce's canonicalize to simplify that process.)

charliermarsh commented 8 months ago

I'd actually expect that virtualenv change to break this case, if I'm reading it correctly:

❯ realpath /opt/homebrew/opt/python@3.8/bin/python3.8
/opt/homebrew/Cellar/python@3.8/3.8.18_2/Frameworks/Python.framework/Versions/3.8/bin/python3.8

So now virtualenv would also use the Cellar path, IIUC?

ofek commented 8 months ago

I don't own a macOS to test that for you unfortunately but that sounds right.

charliermarsh commented 8 months ago

On main, virtualenv uses home = /opt/homebrew/Cellar/python@3.8/3.8.18_2/bin. So, they might have the same problem @cjolowicz?

charliermarsh commented 8 months ago

\cc @gaborbernat -- it seems like virtualenv on main will resolve symlinks for system Pythons, which seems desirable in some cases (hence the issue in virtualenv) but not in others (hence this issue). I can't tell which is "correct" though.

gaborbernat commented 8 months ago

I do not have a good answer here. 😱

konstin commented 8 months ago

We may want to revisit this change (https://github.com/astral-sh/uv/issues/965). I think we should only do this (canonicalize) if we're in a virtual environment?

Agreed, i think this is better, i'm also thinking about the use case where somebody might have intentional redirects for their python setup.

For the Cellar issue, note that technically different patch versions aren't compatible from a packaging perspective, technically a project could require python_full_version != "3.12.2". In practice projects i've only seen lower bounds for patch version (e.g. >3.8.1), upper or exact bounds would be against how python patch versions are maintained and we're building a more reliable tool by intentionally ignoring this detail.

minusf commented 1 month ago

It would be really nice if this was solved somehow, all my uv venvs break after a minor python update...

Besides the difference in the picked executable symlink as described above, I find it interesting that both tools pick the absolute symlink completely differently:

pip_venv/bin/python@ -> python3.12
pip_venv/bin/python3@ -> python3.12
pip_venv/bin/python3.12@ -> /opt/homebrew/opt/python@3.12/bin/python3.12

uv_venv/bin/python@ -> /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/bin/python3.12
uv_venv/bin/python3@ -> python
uv_venv/bin/python3.12@ -> python

FWIW my OCD prefers the pip version, the most specific filename has the absolute symlink (python3.12) instead of uv's least specific (python) 😅

In case it helps anyone here is also the difference between the pyvenv.conf's:

--- pip_venv/pyvenv.cfg 2024-09-13 20:18:47.389618788 +0200
+++ uv_venv/pyvenv.cfg  2024-09-13 20:19:00.968279785 +0200
@@ -1,5 +1,6 @@
-home = /opt/homebrew/opt/python@3.12/bin
+home = /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/bin
+implementation = CPython
+uv = 0.4.9
+version_info = 3.12.6
 include-system-site-packages = false
-version = 3.12.6
-executable = /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/bin/python3.12
-command = /opt/homebrew/opt/python@3.12/bin/python3.12 -m venv /Volumes/sensitive/src/handmade/0/pip_venv
+relocatable = false

charliermarsh commented 1 month ago

I'm open to changing this since it's been reported multiple times. It might be considered breaking though.

frostming commented 1 month ago

I suggest changing it to resolve symbolic links until a non-venv python is found.

lespea commented 1 month ago

I think at the very least it would be nice to have a flag to force one behavior or the other, but maybe there's a concern of bloating commands with too many switches.

charliermarsh commented 1 month ago

I'm fine to change this but it probably requires 0.5.