search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.47k stars 3.01k forks source link

pip install --user fails on OSX-provided Py37 with PEP 517/518 source builds #7922

Open qwhelan opened 4 years ago

qwhelan commented 4 years ago

Environment

Description Building a PEP 517/518 package from source with the --user flag fails due to attempting to install packages in a system location. Error message suggests passing --user flag despite it already being passed:

$ pip install --user --no-binary numpy numpy
Collecting numpy
  Downloading numpy-1.18.2.zip (5.4 MB)
     |████████████████████████████████| 5.4 MB 738 kB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Applications/Xcode_11.2.1/Contents/Developer/usr/bin/python3 /Library/Python/3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-build-env-zwz9998u/overlay --no-warn-script-location --no-binary numpy --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14'
       cwd: None
  Complete output (10 lines):
  Collecting setuptools
    Using cached setuptools-46.1.3-py3-none-any.whl (582 kB)
  Collecting wheel
    Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
  Collecting Cython>=0.29.14
    Downloading Cython-0.29.16-cp37-cp37m-macosx_10_9_x86_64.whl (1.9 MB)
  Installing collected packages: setuptools, wheel, Cython
  ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Python/3.7/site-packages/easy_install.py'
  Consider using the `--user` option or check the permissions.

  ----------------------------------------
ERROR: Command errored out with exit status 1: /Applications/Xcode_11.2.1/Contents/Developer/usr/bin/python3 /Library/Python/3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-build-env-zwz9998u/overlay --no-warn-script-location --no-binary numpy --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.

Expected behavior PEP 517/518 packages should be possible to build from source and installable in a user location

How to Reproduce

  1. On an instance of OSX 10.15, install Python 3.7 is installed: xcode-select --install
  2. Using the system install, attempt to install any PEP 517/518 package with the --user and --no-binary {pkg} flags to ensure we install to a user location and build from source.
  3. You should receive various permission denied errors, likely due to the build environment installation step specifying --no-user

Output

$ pip install --user --no-binary numpy numpy
Collecting numpy
  Downloading numpy-1.18.2.zip (5.4 MB)
     |████████████████████████████████| 5.4 MB 738 kB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Applications/Xcode_11.2.1/Contents/Developer/usr/bin/python3 /Library/Python/3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-build-env-zwz9998u/overlay --no-warn-script-location --no-binary numpy --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14'
       cwd: None
  Complete output (10 lines):
  Collecting setuptools
    Using cached setuptools-46.1.3-py3-none-any.whl (582 kB)
  Collecting wheel
    Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
  Collecting Cython>=0.29.14
    Downloading Cython-0.29.16-cp37-cp37m-macosx_10_9_x86_64.whl (1.9 MB)
  Installing collected packages: setuptools, wheel, Cython
  ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Python/3.7/site-packages/easy_install.py'
  Consider using the `--user` option or check the permissions.

  ----------------------------------------
ERROR: Command errored out with exit status 1: /Applications/Xcode_11.2.1/Contents/Developer/usr/bin/python3 /Library/Python/3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-build-env-zwz9998u/overlay --no-warn-script-location --no-binary numpy --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.

See https://github.com/pydata/bottleneck/issues/338 for initial report

uranusjr commented 4 years ago

Could you try the command with --verbose? pip’s PEP 517 implementation should not install packages to global Python locations at all (because build isolation), and I suspect there is something more involved.

qwhelan commented 4 years ago

Sure:

pip install --user --no-binary numpy numpy --verbose
...
  Given no hashes to check 8 links for project 'Cython': discarding no candidates
  Using version 0.29.16 (newest of versions: 0.29.14, 0.29.15, 0.29.16)
  Collecting Cython>=0.29.14
    Created temporary directory: /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-unpack-o1kl9t50
    Looking up "https://files.pythonhosted.org/packages/66/86/540bf648d0f259e1c59d2710d77f817355a362ed0c8f611d961a33b4ef50/Cython-0.29.16-cp37-cp37m-macosx_10_9_x86_64.whl" in the cache
    Current age based on date: 70948
    Ignoring unknown cache-control directive: immutable
    Freshness lifetime from max-age: 365000000
    The response is "fresh", returning cached response
    365000000 > 70948
    Using cached Cython-0.29.16-cp37-cp37m-macosx_10_9_x86_64.whl (1.9 MB)
    Added Cython>=0.29.14 from https://files.pythonhosted.org/packages/66/86/540bf648d0f259e1c59d2710d77f817355a362ed0c8f611d961a33b4ef50/Cython-0.29.16-cp37-cp37m-macosx_10_9_x86_64.whl#sha256=245e69a1f367c89e3c8a1c2699bd20ab67b3d57053f3c71f0623d36def074308 to build tracker '/private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-req-tracker-kic2mlmo'
    Removed Cython>=0.29.14 from https://files.pythonhosted.org/packages/66/86/540bf648d0f259e1c59d2710d77f817355a362ed0c8f611d961a33b4ef50/Cython-0.29.16-cp37-cp37m-macosx_10_9_x86_64.whl#sha256=245e69a1f367c89e3c8a1c2699bd20ab67b3d57053f3c71f0623d36def074308 from build tracker '/private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-req-tracker-kic2mlmo'
  Installing collected packages: setuptools, wheel, Cython
    Created temporary directory: /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-unpacked-wheel-hwmvu4l3

  ERROR: Could not install packages due to an EnvironmentError.
  Consider using the `--user` option or check the permissions.
  Traceback (most recent call last):
    File "/Library/Python/3.7/site-packages/pip/_internal/commands/install.py", line 404, in run
      use_user_site=options.use_user_site,
    File "/Library/Python/3.7/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
      **kwargs
    File "/Library/Python/3.7/site-packages/pip/_internal/req/req_install.py", line 815, in install
      warn_script_location=warn_script_location,
    File "/Library/Python/3.7/site-packages/pip/_internal/operations/install/wheel.py", line 614, in install_wheel
      warn_script_location=warn_script_location,
    File "/Library/Python/3.7/site-packages/pip/_internal/operations/install/wheel.py", line 414, in install_unpacked_wheel
      clobber(source, lib_dir, True)
    File "/Library/Python/3.7/site-packages/pip/_internal/operations/install/wheel.py", line 392, in clobber
      shutil.copyfile(srcfile, destfile)
    File "/Applications/Xcode_11.2.1/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/shutil.py", line 121, in copyfile
      with open(dst, 'wb') as fdst:
  PermissionError: [Errno 13] Permission denied: '/Library/Python/3.7/site-packages/easy_install.py'
  Cleaning up...
  Removed build tracker: '/private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-req-tracker-kic2mlmo'
  Installing build dependencies ... error
Cleaning up...
  Removing source in /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-install-9ws4sf0h/numpy
Removed numpy from https://files.pythonhosted.org/packages/84/1e/ff467ac56bfeaea51d4a2e72d315c1fe440b20192fea7e460f0f248acac8/numpy-1.18.2.zip#sha256=e7894793e6e8540dbeac77c87b489e331947813511108ae097f1715c018b8f3d from build tracker '/private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-req-tracker-kic2mlmo'
Removed build tracker: '/private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-req-tracker-kic2mlmo'
ERROR: Command errored out with exit status 1: /Applications/Xcode_11.2.1/Contents/Developer/usr/bin/python3 /Library/Python/3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-build-env-fpfgzz5g/overlay --no-warn-script-location -v --no-binary numpy --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/Library/Python/3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/Library/Python/3.7/site-packages/pip/_internal/commands/install.py", line 331, in run
    resolver.resolve(requirement_set)
  File "/Library/Python/3.7/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/Library/Python/3.7/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Library/Python/3.7/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/Library/Python/3.7/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/Library/Python/3.7/site-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/Library/Python/3.7/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/Library/Python/3.7/site-packages/pip/_internal/distributions/sdist.py", line 68, in _setup_isolation
    "Installing build dependencies"
  File "/Library/Python/3.7/site-packages/pip/_internal/build_env.py", line 201, in install_requirements
    call_subprocess(args, spinner=spinner)
  File "/Library/Python/3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Applications/Xcode_11.2.1/Contents/Developer/usr/bin/python3 /Library/Python/3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/wc/18t6w3v130g8tpqh7f6tp4cr0000gn/T/pip-build-env-fpfgzz5g/overlay --no-warn-script-location -v --no-binary numpy --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.

Given the easy_install errors, I checked a couple additional things:

$ python3.7
Python 3.7.5 (default, Oct 22 2019, 10:35:10) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import easy_install
>>> easy_install.__file__
'/Users/chris/Library/Python/3.7/lib/python/site-packages/easy_install.py'

And checking the system location that failed:

$ ls /Library/Python/3.7/site-packages/easy_install.py
ls: /Library/Python/3.7/site-packages/easy_install.py: No such file or directory
uranusjr commented 4 years ago

Do you happen to have pydistutils.cfg somewhere in your system that sets install_lib?

From what I can gather from tracing the code, the problem is that PEP 517 setup tries to do pip install --prefix=<some tmp dir>, but for an unknown reason pip._internal.locations.get_scheme() point it to use /Library/Python/3.7/site-packages for purelib, even though the --prefix option tells it to give something prefixed by that. The only reason I can think of from reading the code is that some distutils config overrides that value. This is very much out of my comfort zone though, I may be entirely off the mark here.

uranusjr commented 4 years ago

Either way, I’m marking this as a bug. If the distutils config is truely the cause, the PEP 517 implementation would need to be amended to ignore it.

qwhelan commented 4 years ago

@uranusjr I'm unfortunately not seeing a pydistutils.cfg on my system.

mgcyung commented 4 years ago

The same problem on Gentoo

pip3 install --user --no-binary pyinstaller pyinstaller
Collecting pyinstaller
  Using cached https://files.pythonhosted.org/packages/3c/c9/c3f9bc64eb11eee6a824686deba6129884c8cbdf70e750661773b9865ee0/PyInstaller-3.6.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.6 /usr/lib64/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0dwcy6c5/overlay --no-warn-script-location --no-binary pyinstaller --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (1 lines):
  ERROR: (Gentoo) Please run pip with the --user option to avoid breaking python-exec
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.6 /usr/lib64/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0dwcy6c5/overlay --no-warn-script-location --no-binary pyinstaller --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.