EnvBuilder and venv symlinks do not work on Windows on 3.7.2

[zooba]

BPO	35854
Nosy	@pfmoore, @vsajip, @tjguk, @ned-deily, @zware, @eryksun, @zooba, @miss-islington
PRs	python/cpython#11700 python/cpython#11700 python/cpython#11700 python/cpython#11707 python/cpython#11707

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/zooba' closed_at = created_at = labels = ['3.7', '3.8', 'OS-windows'] title = 'EnvBuilder and venv symlinks do not work on Windows on 3.7.2' updated_at = user = 'https://github.com/zooba' ``` bugs.python.org fields: ```python activity = actor = 'steve.dower' assignee = 'steve.dower' closed = True closed_date = closer = 'steve.dower' components = ['Windows'] creation = creator = 'steve.dower' dependencies = [] files = [] hgrepos = [] issue_num = 35854 keywords = ['patch', 'patch', 'patch', '3.7regression'] message_count = 10.0 messages = ['334537', '334538', '334545', '334561', '334574', '334596', '334597', '334598', '337721', '337762'] nosy_count = 8.0 nosy_names = ['paul.moore', 'vinay.sajip', 'tim.golden', 'ned.deily', 'zach.ware', 'eryksun', 'steve.dower', 'miss-islington'] pr_nums = ['11700', '11700', '11700', '11707', '11707'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue35854' versions = ['Python 3.7', 'Python 3.8'] ```

[zooba]

The change to pull the redirector executables as part of scripts was... perhaps too clever. There exists code out there that uses EnvBuilder to create environments _without_ copying scripts, which now results in an environment that does not include python.exe.

It also undid symlink support, as scripts are never symlinked.

I'll restore both.

[zooba]

Actually, it seems like venv symlinks are so far from working that they haven't worked (on Windows) since 3.5 or possibly earlier.

I'm just going to make that option ignored and mark it as such in the docs.

[eryksun]

Actually, it seems like venv symlinks are so far from working that they haven't worked (on Windows) since 3.5 or possibly earlier.

I use venv with --symlinks prior to 3.7.2. It works fine as far as I can tell. Perhaps you could elaborate. I assumed you removed it because it's not compatible with the Microsoft Store release for some reason.

The loader doesn't resolve links, so it should work as long as the python.exe, python3x.dll, python3.dll, and vcruntime140.dll files are linked together in the Scripts directory.

    >>> hPython3 = ctypes.WinDLL('python3')._handle
    >>> hVcruntime140 = ctypes.WinDLL('vcruntime140')._handle
    >>> for h in (0, sys.dllhandle, hPython3, hVcruntime140):
    ...     print(_winapi.GetModuleFileName(h), '->')
    ...     print('\t', os.readlink(_winapi.GetModuleFileName(h)))
    ...
    C:\Temp\env36\scripts\python.exe ->
             C:\Program Files\Python36\python.exe
    C:\Temp\env36\scripts\python36.dll ->
             C:\Program Files\Python36\python36.dll
    C:\Temp\env36\scripts\python3.dll ->
             C:\Program Files\Python36\python3.dll
    C:\Temp\env36\scripts\VCRUNTIME140.dll ->
             C:\Program Files\Python36\vcruntime140.dll

    >>> print(sys.executable)
    C:\Temp\env36\scripts\python.exe

>>> print(*sys.path, sep='\n')

C:\Temp\env36\scripts\python36.zip
C:\Program Files\Python36\DLLs
C:\Program Files\Python36\lib
C:\Program Files\Python36
C:\Temp\env36
C:\Temp\env36\lib\site-packages

[zooba]

No, removing it was accidental. But when I tried it myself with a few regular installs of 3.5 and later, none of them updated sys.path correctly, and all of them resolved the links in the loader (or perhaps the shell?).

I hadn't activated the environments, as I generally don't, but if that's required for symlinks to work here then I consider them broken.

[zooba]

Okay, testing more thoroughly on 3.5, symlinks are fine from the console but not via Explorer.

Personally, I dislike that double-clicking python.exe is different from running it from the command line, but so be it. I'll add a note to the docs that this doesn't work.

[eryksun]

Okay, testing more thoroughly on 3.5, symlinks are fine from the console but not via Explorer.

I gave up on using EXE symlinks with Explorer and ShellExecute[Ex]. The shell handles symlinks like shortcuts instead of leaving it up to the file system. In Windows 7, I recall it was thoroughly broken. Symlinks wouldn't even execute. In Windows 10 it's broken in many cases because of the shortcut-like behavior.

I think I know why they're doing this, but I think the fix belongs at a lower level in the system runtime library and loader. It's to accommodate the importance of the application directory. People expect a symlink to an executable to work like a shortcut. In Unix this often just works because most libraries are installed to the system, and an application's private shared libraries can use "$ORIGIN" in the binary's RPATH (or RUNPATH), which refers to the resolved (final) executable directory. In contrast, Windows doesn't use the resolved executable path for the application directory. IMO, they could use a pair of application directories at the start of the search path -- the link's directory and then the resolved executable's directory (or the DLL directory when loading a DLL). Some resources do get resolved like this already. For example, when searching for a "\<local name>\\<exename>.mui" language resource, in Process Monitor we see that it will first try the unresolved application directory (e.g. "C:\Temp\en-US\notepad.exe.mui") and then the resolved path (e.g. "C:\Windows\en-US\notepad.exe.mui").

[zooba]

New changeset a1f9a3332bd4767e47013ea787022f06b6dbcbbd by Steve Dower in branch 'master': bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700) https://github.com/python/cpython/commit/a1f9a3332bd4767e47013ea787022f06b6dbcbbd

[miss-islington]

New changeset 03082a836b707528f885080bda9732d89849d4e3 by Miss Islington (bot) in branch '3.7': bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700) https://github.com/python/cpython/commit/03082a836b707528f885080bda9732d89849d4e3

[ned-deily]

Can we close this issue now?

[zooba]

Yep. Done