"No such file or directory" install error when egg fragment / project name differs from package name

xlqian commented 6 years ago

Pip version: 10.0.0
Python version: 2.7
Operating system: Ubuntu 16.10

Description:

"No such file or directory" when pip-installing with #egg=~

What I've run:

pip install git+https://github.com/clarkduvall/serpy.git@master#egg=TOTO

Errors I've got:

  Running setup.py bdist_wheel for serpy ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lE7moQ/serpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6ovRMI --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-lE7moQ/serpy/setup.py'

  ----------------------------------------
  Failed building wheel for serpy
  Running setup.py clean for serpy
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lE7moQ/serpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-lE7moQ/serpy/setup.py'

  ----------------------------------------

My Guess

It seems that pip will git-clone the project into a tmp dir /tmp/pip-install-lE7moQ/TOTO, but when running the installation, it will look for the setup.py in /tmp/pip-install-lE7moQ/serpy/, which was not produced in pip==9.0.1

dgilland commented 6 years ago

I'm having the same issue as @xlqian when using egg fragments in install URLs using pip@10.0.0. I also confirmed that there's a mismatch for the clone/build directory with the clone target being /tmp/pip-install-<id>/<#egg> but the build target being /tmp/pip-install-<id>/<packagename>.

themmes commented 6 years ago

on OS Ubuntu 16.04LTS, within a conda environment conda 4.5.4
pip 10.0.1 from /home/tom/anaconda3/envs/pcenv/lib/python3.5/site-packages/pip (python 3.5)

I think I am experiencing a similar/same problem, however with an #egg=~ clone link.

Trying to install version of IfcOpenShell.

pip install git+git://github.com/IfcOpenShell/IfcOpenShell.git@v0.5.0-preview2

Returns the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-m9n4zjp9/setup.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-m9n4zjp9/

Sup3rGeo commented 6 years ago

Edit: Not the same problem

francopythonss commented 6 years ago

@themmes I have the same problem too. S.O = Windows10 python = 2.7

I try to install pyqt4 with pip but I get: C:\Users\mario\Downloads\PYQT4>pip2 install pyqt4_gpl_win-4.12.1.zip Processing c:\users\mario\downloads\pyqt4\pyqt4_gpl_win-4.12.1.zip Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in IOError: [Errno 2] No such file or directory: 'c:\users\mario\appdata\local\temp\pip-req-build-ruppnu\setup.py'

uranusjr commented 6 years ago

I just tested this, and the complete is actually much more confusing than provided in the top post (the same from 10.0 to master):

Collecting TOTO from git+https://github.com/clarkduvall/serpy.git@master#egg=TOTO
  Cloning https://github.com/clarkduvall/serpy.git (to revision master) to /private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-install-uSNmoa/TOTO
  Running setup.py (path:/private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-install-uSNmoa/TOTO/setup.py) egg_info for package TOTO produced metadata for project name serpy. Fix your #egg=TOTO fragments.
Requirement already satisfied: six in /Users/uranusjr/Documents/programming/python/pip/venv-2.7/lib/python2.7/site-packages (from serpy) (1.11.0)
Building wheels for collected packages: serpy, serpy
  Running setup.py bdist_wheel for serpy ... done
  Stored in directory: /private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-ephem-wheel-cache-gwh7yk/wheels/c8/b1/cb/7ffff1a5befd122710f4f588c539c82859a91cffb296786f91
  Running setup.py bdist_wheel for serpy ... error
  Complete output from command /Users/uranusjr/Documents/programming/python/pip/venv-2.7/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-install-uSNmoa/serpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-wheel-p7Xw9c --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-install-uSNmoa/serpy/setup.py'

  ----------------------------------------
  Failed building wheel for serpy
  Running setup.py clean for serpy
  Complete output from command /Users/uranusjr/Documents/programming/python/pip/venv-2.7/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-install-uSNmoa/serpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/private/var/folders/sg/5486g341789bflp89sn_3qcc0000gn/T/pip-install-uSNmoa/serpy/setup.py'

  ----------------------------------------
  Failed cleaning build dir for serpy
Successfully built serpy
Failed to build serpy
Installing collected packages: serpy
Successfully installed serpy-0.3.1

Some initial observations:

The installation actually succeeds in the end (!)
There is a warning pinpointing the underlying cause: #egg= must match the project name in metadata
The error seems to come from the setuptools shim
Installed package uses the name specified in metadata, not #egg= (which is only used to name the source directory)

uranusjr commented 6 years ago

Also @francopythonss your error is caused by a different code path, and probably (I am not sure) should be reported as a separate issue. This issue is specific to installing from VCS with #egg=.

uranusjr commented 6 years ago

I did some debugging, here’s what I observed:

The package in question (serpy in this case) is actually installed twice. As mentioned previously, both are under the same name serpy.

The first installation correctly succeeds, but the second one generates the error because the source is deleted during the clean up phase of the first installation.

The second installation entry is added by the resolver. pip install generates a Requirement object from the argument passed to it, and adds it to a RequirementSet, which keeps Requirement objects as an OrderedDict (and other things not relevant here). This RequirementSet is passed to the resolver, which recursively discovers more dependencies, and adds more Requirement objects for those dependencies to the RequirementSet. The mismatching #egg= value, however, makes RequirementSet keep the initial Requirement under a wrong key (TOTO) during initialisation. So when the resolver processes the Requirement of serpy, it incorrectly thought it is not in the RequirementSet yet, and adds it again, this time under the correct key:

https://github.com/pypa/pip/blob/116c3b1c7ec7970481b65b886ec900c5b54f2c30/src/pip/_internal/resolve.py#L290-L297

Unfortunately, after knowing this, the fix is still not straightforward. For a remote, non-dist requirement, pip cannot know the package name before it downloads and runs setup.py (which is why #egg= is needed), but it cannot run setup.py before it knows what the package is. [] So the #egg= specification (or the name before @ for the PEP 508 variant) really really must* match the one in setup.py. So maybe the “fix” would be to hard-error when pip finds they don’t match, instead of showing a warning and failing later down the line.

[*]: Theoretically I believe it can, but that would probably require a huge restructure to implement.

cjerdonek commented 6 years ago

I don't know if hard-erroring is permissible or not (e.g. for backwards compatibility). However, as long as pip isn't hard erroring, what about fixing the RequirementSet key once pip notices it is wrong? It seems that has the potential of letting pip continue without doing more harm.

uranusjr commented 6 years ago

It is definitely possible. pip can already perform some name inference if you don’t provide the #egg= at all, in fact. This works correctly:

pip install git+https://github.com/clarkduvall/serpy.git@master

But I don’t know exactly when this would work and when not. The mismatching name causes inconsistencies within the data structure, and while this particular RequirementSet problem is relatively easy to plug, I don’t know if there are more inconsistencies that can pop up in the future (do anyone?), and what consequences they would cause. Personally I think it would make pip harder to maintain in the future.

Maybe a possible approach is to hard-error on mismatching egg name, and instruct the user to try removing it. This would mostly solve the problem if the egg name inference works most of the time (but again, I don’t know if it does).

cjerdonek commented 6 years ago

I don’t know if there are more inconsistencies that can pop up in the future

Yes, that is definitely possible. If a change to fix the key were done, one would definitely want to audit / review the code to make sure no code still has references to the old name (and possibly do some refactoring to make sure the name is always being accessed through a single code path).

Issue #5051 will help with making the installation code simpler to reason about (perhaps more functional), but that is a longer term project.

saif-ellafi commented 5 years ago

I was having this issue until I noticed I should have quoted the whole thing in command line

e.g. pip install "git+https://github.com/clarkduvall/serpy.git@master#egg=TOTO"

ncoghlan commented 5 years ago

From the report and discussion in https://github.com/pypa/packaging-problems/issues/252, it looks like projects switching over to PEP 517 from build_requires in setuptools may start encountering this bug, as one of the effects of that transition is that they go from installing their build dependencies with easy_install to installing them with pip, and the two installers behave differently in this regard.

rbhanot-1mg commented 4 years ago

This error drove me crazy since the name of the package was different in setup.py and the name mentioned in egg. It was not really practical for me to change all the egg names to match the name in setup.py. I upgraded my pip version to 9.0.3 and this fixed my issue.

dHannasch commented 4 years ago

This error can also be caused by a dependency specified in setup.py install_requires like 'sk-video @ git+https://github.com/scikit-video/scikit-video.git@master', which appears to be equivalent to to how you'd specify egg=sk-video when pip-installing on the command line.

With such a setup.py:

$ pip --version
pip 19.3.1
$ pip install --editable .
Building wheels for collected packages: scikit-video, scikit-video
  Building wheel for scikit-video (setup.py) ... done
  Created wheel for scikit-video: filename=scikit_video-1.1.11-cp27-none-any.whl size=77817 sha256=8e647a89e5a8f95818d981b842c9bd0e78f02b89ec8ae196d6c2adcf4626570a
  Stored in directory: /tmp/pip-ephem-wheel-cache-UUunUy/wheels/1f/3a/a4/5887855bd98bc84c9e526fff6ced51eaa576cbce91796bc4f4
  Building wheel for scikit-video (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/david/anaconda/envs/tmpEnv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1I2fPi/scikit-video/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1I2fPi/scikit-video/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-n4YV__ --python-tag cp27
       cwd: /tmp/pip-install-1I2fPi/scikit-video/
  Complete output (3 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-1I2fPi/scikit-video/setup.py'
  ----------------------------------------
  ERROR: Failed building wheel for scikit-video
  Running setup.py clean for scikit-video
  ERROR: Command errored out with exit status 1:
   command: /home/david/anaconda/envs/tmpEnv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1I2fPi/scikit-video/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1I2fPi/scikit-video/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: /tmp/pip-install-1I2fPi/scikit-video
  Complete output (3 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-1I2fPi/scikit-video/setup.py'
  ----------------------------------------
  ERROR: Failed cleaning build dir for scikit-video
Successfully built scikit-video
Failed to build scikit-video

Fortunately, when installing directly from GitHub, you're not bound to use the same name that PyPI uses, so in this example you can just change it to 'scikit-video @ git+https://github.com/scikit-video/scikit-video.git@master'.

pradyunsg commented 4 years ago

We'd probably want to look into this as apart of #6607.

dove-young commented 4 years ago

Found similar issue when install retrying package via pip.

/usr/bin/python3.6 -m pip install --no-clean retrying

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-x9jl6v24/retrying/setup.py", line 32, in <module>
        with open('README.rst') as file_readme:
    FileNotFoundError: [Errno 2] No such file or directory: 'README.rst'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

pypa / pip