eryksun
commented
1 year ago When Nushell resolves a command name from a relative path that has one or more slashes or backslashes, such as ".venv/Scripts/python.exe ", it seems to always create an extended path, i.e. a path that's prefixed by "\\?\". This gets set as the lpApplicationName parameter when the shell calls CreateProcessW(). This in turn is the result of calling GetModuleFileNameW(NULL, ...) in the spawned process.
Example in Nu:
C:\Program Files〉Python310\python.exePython 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, _winapi
>>> sys.orig_argv
['Python310\\python.exe']
>>> _winapi.GetModuleFileName(0)
'\\\\?\\C:\\Program Files\\Python310\\python.exe'
>>> sys.executable
'C:\\Program Files\\Python310\\python.exe'Prior to Python 3.11, GetModuleFileNameW() is called at startup in order to get the path to set as sys.executable. So the question is, why isn't sys.executable an extended path in the above example? What's happening is that in 3.6 to 3.10, the initialization code in "PC/getpathp.c" converts an extended path into a regular path via WinAPI PathCchCanonicalizeEx().
Python 3.11 is the first version on Windows that sets sys.executable based on the first command-line argument (i.e. argv[0]) instead of basing it on GetModuleFileNameW(NULL, ...). It still falls back on on the latter, but only if the resolved argv[0] path doesn't exist. Usually argv[0] from the command line is either a regular absolute path or a relative path, in which case, if the path exists, sys.executable will also be a regular path.
On the other hand, when running from a virtual environment that uses a launcher, the value of sys.executable gets passed to the base Python process in the "__PYVENV_LAUNCHER__" environment variable. The venv launcher gets this value from calling GetModuleFileNameW() instead of from the command line. The problem is that the new implementation of "Modules/getpath.c" in 3.11 doesn't canonicalize the path. Thus sys.executable ends up being an extended path in this case.
Note that if you create the environment with symlinks instead of using a launcher, then sys.executable is based on argv[0]. For example:
C:\Temp〉py -3.11 -m venv --symlinks env311
C:\Temp〉env311\Scripts\python.exePython 3.11.2 (tags/v3.11.2:878ead1, Feb 7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, _winapi
>>> sys.orig_argv
['env311\\Scripts\\python.exe']
>>> _winapi.GetModuleFileName(0)
'\\\\?\\C:\\Temp\\env311\\Scripts\\python.exe'
>>> sys.executable
'C:\\Temp\\env311\\Scripts\\python.exe'I suppose the simplest fix would be to update the venv launcher to canonicalize the executable path in its get_process_name() function.
A more reliable fix would update "Modules/getpath.c" to always canonicalize the executable path, regardless of how it's determined.
Because of this, some packages cannot be installed with pip. And everything is broken which needs to have a valid
sys.executable.
Can you provide some example packages that fail to install or misbehave in this case?
An extended path is a valid file path, valid as the process executable path, and valid in most other cases. However, an extended path should not be set as the current working directory. The Windows API doesn't support that.
presidento
zooba
stefaneidelloth
Bug report
Minimal reproducible example in Windows:
Your environment
Nushell 0.76.0, Windows 10 64bit. It works perfectly with system Pythons (no virtualenv), or in virtualenv except for Python 3.11 (this is why I think it is something related to Python, and not only to Nushell):
Because of this, some packages cannot be installed with
pip. And everything is broken which needs to have a validsys.executable.