search

python / cpython

The Python programming language
https://www.python.org
Other
63.68k stars 30.51k forks source link

venv: optional activate link/copy to solve ci/portability headaches #100281

Open kfsone opened 1 year ago

kfsone commented 1 year ago

Problem

activate location is platform dependent.

Background

15+ years ago PyPY made a decision that has often been a headache for portability; putting activate in Scripts for Windows and bin for everyone else.

This is sometimes misreported as a bug (https://github.com/python/cpython/issues/97586), core issue (https://bugs.python.org/issue35003), discussion (https://github.com/python/cpython/pull/18083) and recurrent threads on the pypy issues list (https://github.com/pypa/virtualenv/issues/2404, https://github.com/pypa/virtualenv/issues/1418#issuecomment-910044412)

It's an exceptional pain point because 1/ you chose python to ease portability, -2->/ you use venv as an extra layer of abstraction, -3->/ surprise! your abstraction requires you to take two different routes based on platform!

Proposal

Introduce an option that causes the creation of a symlink or second copy of the activator at a user specified path.

Ubuntu|Pwsh> venv -p python 3.10 --activator-copy=/opt/ci/activate /opt/ci
Windows|Pwsh> venv -p python 3.10 --activator-copy=o:/ci/activate o:/ci

Arguments

(Per the original change) "Scripts is the correct location for activate on Windows"

It's not a big deal, just if .. then else

CI and building today is complex with many moving parts, having an intermediate level abstraction suddenly require you to think about platforms is always a headache. If you're using something like gitlab, you end up with a huge hitch in complexity to achieve this for the scope it's in under the current system.

Windows doesn't support symlinks

Whether it is a file or a symlink is irrelevant, and anyone who is actively working to support a version of Windows old enough to not support symlinks already has enough of their hair on fire that they'll gladly accept it falling back to creating a file instead of junction.

eryksun commented 1 year ago

This is sometimes misreported as a bug (https://github.com/python/cpython/issues/97586)

I changed #97586 into a feature request a couple months ago. Creating a relative symlink or a junction (bind mount) to the "Scripts" directory is a reasonable approach if the filesystem supports reparse points (e.g. NTFS and ReFS). The downside is that there's no fallback if reparse points aren't supported (e.g. FAT32, exFAT and most UNC providers, such as a VirtualBox shared folder).

Your proposal to symlink or copy just the activation script will require modifying the PowerShell script "Activate.ps1". Currently it's implemented to be relative to the directory of the shell script. Thus a symlink or copy in "bin" would add "bin" to PATH instead of "Scripts".

Python itself now violates the Scripts rule

Putting the base interpreter binaries in "Scripts" makes about as much sense, literally speaking, as putting scripts in "bin", so I'd be fine with making that change to the base installation layout.

15+ years ago PyPY made a decision that has often been a headache for portability; putting activate in Scripts for Windows and bin for everyone else.

Using "Scripts" on Windows dates back to the mid 1990s. Here's what this looked like in distutils when the platform install schemes were added in 2000.

https://github.com/python/cpython/blob/a233d86638efc76efeddb40e38a844b97c35f2a4/Lib/distutils/command/install.py#L15-L40

kfsone commented 1 year ago

Your proposal to symlink or copy just the activation script will require modifying the PowerShell script "Activate.ps1". Currently it's implemented to be relative to the directory of the shell script. Thus a symlink or copy in "bin" would add "bin" to PATH instead of "Scripts".

I'm not suggesting that, I'm suggesting that the user be allowed to specify their own location entirely. The tldr of this proposal is Introduce an option that causes the creation of a symlink or second copy of the activator at a user specified path.

"--activator-copy : For this invocation of the command only, a copy of the activator will be created with this filename, or a symlink/junction where available. This path is not kept anywhere by Python so it is the user's responsibility to update/remove it or include it in their PATHs."

edit: "created" - I leave whether it should be obstructive or overwritable open for consideration.

eryksun commented 1 year ago

A symlink or copy of "Activate.ps1" in an arbitrary directory will not work correctly without making changes to how "Activate.ps1" is implemented. As is, it would add the directory where the symlink or copy exists to PATH instead of the "Scripts" directory of the environment.