No way to create 'copy' virtualenvs on MacOS

[craigds]

Issue

The docs specify a --copies option which should copy the Python interpreter when creating a virtualenv.

This option is not recognised on MacOS.

The stdlib venv module is happy to accept the --copies option (and it works), so this is a shortcoming of the virtualenv package.

Environment

• virtualenv 20.26.1 from /opt/homebrew/lib/python3.12/site-packages/virtualenv/__init__.py
• OS: MacOS 14.4.1
• pip list of the host python where virtualenv is installed:

WARNING: Skipping /opt/homebrew/lib/python3.12/site-packages/packaging-24.0.dist-info due to invalid metadata entry 'name'
Package                 Version
----------------------- --------
certifi                 2024.2.2
cffi                    1.16.0
cfgv                    3.4.0
distlib                 0.3.8
filelock                3.13.4
GDAL                    3.8.5
gpg                     1.23.2
identify                2.5.35
nodeenv                 1.8.0
numpy                   1.26.4
openvino                2024.0.0
pip                     24.0
platformdirs            4.2.0
pre-commit              3.7.0
pycparser               2.21
PyQt3D                  5.15.6
PyQt5                   5.15.10
PyQt5-sip               12.13.0
PyQt6                   6.6.1
PyQt6-3D                6.6.0
PyQt6-Charts            6.6.0
PyQt6-DataVisualization 6.6.0
PyQt6-NetworkAuth       6.6.0
PyQt6-sip               13.6.0
PyQt6-WebEngine         6.6.0
PyQtChart               5.15.6
PyQtDataVisualization   5.15.5
PyQtNetworkAuth         5.15.5
PyQtPurchasing          5.15.5
PyQtWebEngine           5.15.6
PyYAML                  6.0.1
setuptools              69.0.2
TBB                     0.2
typing_extensions       4.8.0
virtualenv              20.26.1
wheel                   0.43.0
WARNING: Skipping /opt/homebrew/lib/python3.12/site-packages/packaging-24.0.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /opt/homebrew/lib/python3.12/site-packages/packaging-24.0.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /opt/homebrew/lib/python3.12/site-packages/packaging-24.0.dist-info due to invalid metadata entry 'name'

Output of the virtual environment creation

$ virtualenv --copies -vvv --with-traceback x
67 setup logging to NOTSET [DEBUG report:36]
72 find interpreter for spec PythonSpec(path=/opt/homebrew/opt/python@3.12/bin/python3.12) [INFO builtin:72]
73 filesystem is not case-sensitive [DEBUG info:25]
73 proposed PythonInfo(spec=CPython3.12.3.final.0-64, exe=/opt/homebrew/opt/python@3.12/bin/python3.12, platform=darwin, version='3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 15.0.0 (clang-1500.3.9.4)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:79]
73 accepted PythonInfo(spec=CPython3.12.3.final.0-64, exe=/opt/homebrew/opt/python@3.12/bin/python3.12, platform=darwin, version='3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 15.0.0 (clang-1500.3.9.4)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:81]
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-mac-brew,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: unrecognized arguments: --copies

[gaborbernat]

That flag only exists if your platform supports copy. There must be some kind of issue on your machine that our copy detection fails. PR welcome.

[craigds]

Doesn't every filesystem support copy? I'm not sure what that means to be honest.

Can you point me to the code where this detection occurs?

[gaborbernat]

You can only copy something if you can read it https://github.com/pypa/virtualenv/blob/main/src/virtualenv/create/via_global_ref/builtin/ref.py#L64

[craigds]

That makes sense at runtime where the question is "can I copy this specific file?" But it seems that virtualenv is making this decision gloablly before even adding the option to the arg parser. Any idea why that could happen?

[gaborbernat]

We don't want to expose flags that cannot work. If you read the code you will see that the flag is only added if the copy is possible.

[craigds]

I get that. I'm confused though. Your first reply says that the copy detection is based on whether the specific file you're trying to copy can be copied.

But in fact, --copies isn't even in the --help info on MacOS. So, I wonder what it's actually based on?

I'm happy to spend some time on this today but I wanted to understand if you were aware of any reasons this could be.

(This is a bug one way or another - copy is possible because python -m venv --copies works fine without issue on this same machine)

[craigds]

Okay some debugging points to this being caused by homebrew themselves.

> /Users/cdestigter/code/virtualenv/src/virtualenv/create/via_global_ref/api.py(24)can_copy()
-> return not self.copy_error
(Pdb) self.copy_error
'Brew disables copy creation: https://github.com/Homebrew/homebrew-core/issues/138159'

Rather frustrating; the reason I want to use copies is to prevent homebrew from breaking my virtualenvs every time they upgrade python.

I'll raise this issue there though. Thanks for your replies

[gaborbernat]

You're welcome.

[gaborbernat]

You probably should read https://justinmayer.com/posts/homebrew-python-is-not-for-you, I recommend using pyenv instead for Python development.