Build isolation doesn't work with Python installation that requires PYTHONPATH for stdlib

[vidartf]

Environment

• pip version: 19.2.3
• Python version: 2.7 / 3.7
• OS: Windows (will probably fail on nix as well?)

Custom env, with PYTHONPATH, PYTHONHOME set.

Description When installing PEP518/517 package based on setuptools, the install fails with an ImportError: No module named site on py 2.7, or ModuleNotFoundError: No module named 'encodings' on py3.7.

Expected behavior Install completes successfully.

How to Reproduce Repro on Windows:

1. Take a normal/fresh CPython install with pip installed in e.g. C:\testpython, put it in PATH (without any other py installs) (SET PATH=C:\testpython).
2. SET PYTHONHOME=C:\testpython (to ensure no other system python is picked up via registry).
3. Move C:\testpython\Lib to C:\testpython\hidden\Lib.
4. See that calling python fails with the above error.
5. SET PYTHONPATH=C:\testpython\hidden\Lib;C:\testpython\hidden\Lib\site-packages.
6. See that python now works again
7. Upgrade pip if needed with python -m pip install --upgrade pip --target=C:\testpython\hidden\Lib\site-packages
8. Do a source install of a a package with pyproject.toml that uses setuptools under the hood. python -m pip install path_to_src_package --target=C:\testpython\hidden\Lib\site-packages. It fails since pip internally overwrites PYTHONPATH for the build env, so its subprocess call to sys.executable fails. This is where it overwrites PYTHONPATH:

https://github.com/pypa/pip/blob/53ee380d95b49a66c87eea6bda41b1c1fef7c600/src/pip/_internal/build_env.py#L123-L127

Output

C:\testpython>python -m pip install path_to_src_package
Processing path_to_src_package
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 3221226505:
   command: 'C:\pytest\python.exe' 'C:\testpython\hidden\Lib\site-packages\pip\_vendor\pep517\_in_process.py' get_requires_for_build_wheel 'C:\Users\username\AppData\Local\Temp\tmpr4n0ozpb'
       cwd: C:\Users\username\AppData\Local\Temp\pip-req-build-o60yaw8r
  Complete output (4 lines):
  Fatal Python error: initfsencoding: unable to load the file system codec
  ModuleNotFoundError: No module named 'encodings'

  Current thread 0x000027b8 (most recent call first):
  ----------------------------------------
ERROR: Command errored out with exit status 3221226505: 'C:\testpython\python.exe' 'C:\testpython\hidden\Lib\site-packages\pip\_vendor\pep517\_in_process.py' get_requires_for_build_wheel 'C:\Users\username\AppData\Local\Temp\tmpr4n0ozpb' Check the logs for full command output.

[chrahunt]

Hello. Thanks for the easy reproduction instructions and clear report.

What is the use case for a Python that only functions with a specially set PYTHONPATH?

[vidartf]

What is the use case for a Python that only functions with a specially set PYTHONPATH?

This would be an issue e.g. when the dist follows the *nix folder layout for all platforms. You could get such a case to work by doing a custom build of CPython for Windows that looks in lib/pythondist instead of Lib, but using a envvar is of course a lot less work.

[pfmoore]

IMO, this is sufficiently obscure that it's not something pip should try to handle. Given that by default pip doesn't install in the correct place anyway (note that --target is needed to install to the correct directory), there are going to be many features of pip that don't work as expected (upgrades, --user, etc).

It may be worth pip explicitly catching this situation (if that's practical to do) and report an error saying that this is an unsupported configuration, but I doubt that it's something we should try to support directly.

[chrahunt]

I agree that this isn't something we should support. If the non-standard paths are a hard requirement then a custom build is probably the way to go (acknowledging that this is more work, as mentioned above). Fundamentally if we can't rely on the Python being self-sufficient then build isolation gets a lot harder, and we'd need to expose essentially all aspects of it for end user configuration.

What would we catch here, that the standard library is importable when isolated? I'm torn since this would probably require another subprocess, but would be very unlikely to happen in practice. It would not be worth the overhead, IMO.

[vidartf]

Sure, resolving this as wontfix is ok. The main point for me was to have a ruling on it in case there was a smart fix that was not liable to break stuff.

For completeness, I've been wondering if setting the PythonPath registry keys according to https://docs.python.org/2/using/windows.html#finding-modules / https://docs.python.org/3.7/using/windows.html#finding-modules might solve this? It isn't clear to me from the docs if all such PythonPath keys are added to sys.path (and if so, what the order is), or if only some entries are added? If this is not done correctly, I'm assuming that unsetting PYTHONPATH might make the isolated build env end up with an stdlib from another Python installation?

[pfmoore]

I've been wondering if setting the PythonPath registry keys might solve this?

Possibly, but I'd still consider that to be an entirely unsupported configuration, so I'd say give it a try, and if it gives you a way forward, go for it. All I'd ask is that if at any point, you hit any pip issues, you demonstrate that they happen in a vanilla environment before asking for help here. (Or at a minimum, you make it clear that you've got a weird setup and link back here in the issue report).

[vidartf]

All I'd ask is that if at any point, you hit any pip issues, you demonstrate that they happen in a vanilla environment before asking for help here. (Or at a minimum, you make it clear that you've got a weird setup and link back here in the issue report).

Of course. I was hoping the OP was proof of me understanding what a proper MWE is 😉

[pfmoore]

Of course. I was hoping the OP was proof of me understanding what a proper MWE is 😉

Sorry, I wasn't trying to imply otherwise. Thanks for making this issue really easy to diagnose.

[pradyunsg]

Thanks @vidartf for a basically excellent bug report; and for being understanding on why we view this as a wont-fix.

I'll go ahead and close this now. :)