ensurepip and pip install failures on macOS Sierra with non-system Python 2.7.x

[4e6c65ea-92c6-4973-8392-4f4424b02e32]

BPO	28440
Nosy	@ronaldoussoren, @benjaminp, @ned-deily
PRs	python/cpython#5445
Files	pipfails.patch: patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/ned-deily' closed_at = None created_at = labels = ['OS-mac', '3.7'] title = 'ensurepip and pip install failures on macOS Sierra with non-system Python 2.7.x' updated_at = user = 'https://bugs.python.org/MarcCuller' ``` bugs.python.org fields: ```python activity = actor = 'ned.deily' assignee = 'ned.deily' closed = False closed_date = None closer = None components = ['macOS'] creation = creator = 'Marc.Culler' dependencies = [] files = ['45090'] hgrepos = [] issue_num = 28440 keywords = ['patch'] message_count = 8.0 messages = ['278649', '278657', '279779', '282272', '282275', '311261', '347833', '347840'] nosy_count = 5.0 nosy_names = ['ronaldoussoren', 'benjamin.peterson', 'ned.deily', 'python-dev', 'Marc.Culler'] pr_nums = ['5445'] priority = None resolution = None stage = 'patch review' status = 'open' superseder = None type = None url = 'https://bugs.python.org/issue28440' versions = ['Python 2.7', 'Python 3.6', 'Python 3.7'] ```

[4e6c65ea-92c6-4973-8392-4f4424b02e32]

Changes made to /Library/Python on macOSX Sierra cause the --with-ensurepip compiler flag to fail, and lead to failures of pip after installing Python.

The new file that causes the problem on Sierra is: /Library/Python/2.7/site-packages/Extras.pth The current version of site.py reads that .pth file, which results in sys.path containing the path: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python The latter directory (which is protected by SIP) contains many Python 2.7 packages, including easy_install, setuptools, six, py2app, numpy, pylab and pyOpenSSL. The effect of including this SIP-protected path in sys.path is this: installing or upgrade a package in /Library/Frameworks/Python.framework which is also installed as an "Extra" in the /System/Frameworks/Python.framework will cause pip to attempt to delete the "old" package from its SIP-protected directory, leading to a "Permission Denied" exception and a failed install.

Given that Apple has now tied /Library/Python to the system Python in this way, thereby making a separate Python framework in /Library/Frameworks unusable, the natural solution to this problem would be to stop including any /Library/Python paths in sys.path.

I am attaching a patch that removes the block of code in site.py which adds these paths.

[ned-deily]

This also affects the -m ensurepip installation of pip itself: the pip install fails trying to upgrade the Apple-supplied version of setuptools. The behavior of adding the system Python's site-packages directory to the search path of all framework-build Pythons was deliberately added as a result of bpo-4865. I think experience has shown this was not a good idea because of the coupling it introduced between separate Python installations and some third-party distributors of Python on OS X already patch this code out. It's currently only an issue for 2.7.x since Apple has not yet shipped versions of Python 3.x but the code should be removed there, too.

[ronaldoussoren]

Another reason to remove this feature: installing python 2.7.12 using the installer on www.python.org breaks the system install of Python, likely because of this issue.

I'm a bit sad that this feature has to go, but modern Python use (in particular virtualenv) has reduced the need for this feature.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset a8a342b3fbc7 by Ned Deily in branch '2.7': Issue bpo-28440: No longer add /Library/Python/site-packages, the Apple-supplied https://hg.python.org/cpython/rev/a8a342b3fbc7

[ned-deily]

I've pushed the change to revert bpo-4865 for the 2.7 branch for release with 2.7.13. I am leaving this open to push similar changes for 3.x branches in anticipation of Apple someday supplying system Python 3.x. It would also be good to see if the post-install script that runs ensurepip can be made to fail more noisily.

[ned-deily]

New changeset 763f094b1f0ce2a07768828e78afe9687e9ad3bb by Ned Deily in branch 'master': bpo-28440: Don't add /Library/Python/3.x/site-packages to sys.path (bpo-5445) https://github.com/python/cpython/commit/763f094b1f0ce2a07768828e78afe9687e9ad3bb

[ronaldoussoren]

In response to msg282272: That day has come, the beta for 10.15 contains Python 3.7.3.

[ned-deily]

That day has come, the beta for 10.15 contains Python 3.7.3.

Yay! FTR, 763f094b1f0ce2a07768828e78afe9687e9ad3bb was released in 3.7.0.

The only reason I have left this issues open was that I also wanted to make sure that ensurepip failures in the post-install script do not fail silently.

[ronaldoussoren]

What's left to do for this issue?