pip install -U setuptools fails in fresh conda environment

[stxlvt]

Setup:

D:\workspace>conda info
Current conda install:

             platform : win-32
        conda version : 3.18.5
  conda-build version : 1.14.1
       python version : 2.7.10.final.0
     requests version : 2.8.1
     root environment : D:\anaconda32  (writable)
  default environment : D:\anaconda32
     envs directories : D:\anaconda32\envs
        package cache : D:\anaconda32\pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/win-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

D:\workspace>conda create --name tst python
Fetching package metadata: ....
Solving package specifications: ............
Package plan for installation in environment D:\anaconda32\envs\tst:

The following NEW packages will be INSTALLED:

    msvc_runtime: 1.0.1-vc9_0   [vc9]
    pip:          7.1.2-py27_0
    python:       2.7.10-4
    setuptools:   18.4-py27_0
    wheel:        0.26.0-py27_1

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate tst
#

D:\workspace>activate tst
Activating environment "D:\anaconda32\envs\tst"...

The bug:

[tst] D:\workspace>pip install -U setuptools
Collecting setuptools
  Downloading setuptools-18.5-py2.py3-none-any.whl (462kB)
    100% |################################| 462kB 787kB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 18.4
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

Trying to install -U any package depending on setuptools also fails with the same error.

Known workarounds:

• Adding the --ignore-installed option will sucessfully upgrade setuptools and the problem will be solved for the lifetime of that particular conda environment.
• Downloading and running ez_setup.py also fixes the problem for that conda environment.

The error message is correct, there is no easy-install.pth file in site-packages. The error occurs when pip tries to uninstall the old setuptools package, in pip/req/req_uninstall.py:160.

The second workaround will create the missing pth file, while the first will not, but in both cases everything seems to work afterwards.

I am unsure if this is a pip issue or a conda issue, but because it is your setuptools build that makes pip choke, I choose to submit it here.

[adamdivak]

Thank you for the report, I just hit the same issue and could solve it using the --ignore-installed option.

[ccordoba12]

I'd recommend you to not upgrade setuptools using pip. It's better to wait for us to provide a new package because (as you noticed) we don't use easy-install.pth.

I think that was decided to avoid setuptools messing with sys.path (which is one of its least appreciated features :-)

[stxlvt]

Thank you for your reply, ccordoba12. The problem with the "then don't do that" advice is that this bug also hits you if you try to upgrade a package that depends on setuptools. For example:

[tst] D:\workspace>pip install -U zc.buildout
Collecting zc.buildout
  Downloading http://python/root/pypi/+f/739/582d22e3ddd5e/zc.buildout-2.5.0-py2
.py3-none-any.whl (261kB)
    100% |################################| 262kB 6.8MB/s
Collecting setuptools>=8.0 (from zc.buildout)
  Downloading http://python/root/pypi/+f/d40/182384798e286/setuptools-18.8.1-py2
.py3-none-any.whl (463kB)
    100% |################################| 466kB 10.2MB/s
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 18.5
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

To try to figure out how big a problem this is, I downloaded the file pypi-deps.csv.lzma from here, and it showed that at the time that data was compiled in 2013, 1822 pypi packages depended on setuptools, and a further 41 depended on one of those 1822 packages. That data is used in this zoomable graph of pypi dependencies. Setuptools is the red dot in the middle of the biggest blob. It seems like no other package is depended on as much as setuptools.

I can totally understand not wanting to mess with sys.path. Been there, done that. But I had hoped you could find a way to solve this that does not break updating a big portion of the packages on pypi.

[ccordoba12]

Pinging @ilanschnell about this one.

[ilanschnell]

Use conda to update setuptools:

conda update setuptools

Using pip, I get the same error:

Cannot remove entries from nonexistent file /Users/ilan/anaconda/ibuild/tmp/lib/python2.7/site-packages/easy-install.pth

The file pip is trying to remove (easy-install.pth) does not exist in a fresh Anaconda/Miniconda installation because no conda packages are able to include this file. I consider it a pip (or setuptools) bug that pip install -U setuptools fails in the absence of this file.

[bkreider]

@kalefranz Can you look at this? It's still an issue (as seen on the mailing list)

It's tied to this issue over at pip: https://github.com/pypa/pip/issues/2751#issuecomment-165541708

So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well.

It's true. The conda package for setuptools installs an egg.

[den-run-ai]

this worked for me perfectly on Windows (original recipe for OSX):

https://github.com/tensorflow/tensorflow/issues/135#issuecomment-174325470

curl https://bootstrap.pypa.io/ez_setup.py -o - | python

[burkesquires]

The --ignore-installed helped to get past the error for upgrading to iPython 5.0.

I was getting the following error when running pip install ipython --upgrade

Installing collected packages: setuptools, traitlets, ipython Found existing installation: setuptools 23.0.0 Cannot remove entries from nonexistent file /Users/squiresrb/anaconda/lib/python3.5/site-packages/easy-install.pth

Thanks!

[stxlvt]

@ilanschnell @ccordoba12

Please consider reopening this bug, or at least responding to pypa/pip#2751 which awaits an answer from you guys. This problem is not going away, and there is a constant stream of references from other projects who have this problem, both here and at pypa/pip#2751. And that's only the tip of the iceberg, those that do not just find this workaround and apply it without telling us about it.

First step to solving this problem is that you guys and the pip guys must determine whose problem it is. :-)

To bring you up to speed on where things stand I'll give you a summary. @rbtcollins pointed out that...

The assumption of easy-install.pth file is guarded by a check for .egg. So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well. Perhaps don't do that?

To which @ilanschnell answered:

Correct, conda packages usually include .egg-info [...]

And @rbtcollins replied:

No, you've misunderstood. .egg-info is not the same as .egg. The test I pasted is for "SOMETHING.egg", not "SOMETHING.egg-info".

So. Could you perhaps make setuptools look more like the pip setuptools by making it a normal package instead of an egg? Would that be an acceptable solution?

[ssbarnea]

Can we have an update on this? This bg break tons of packages making impossible to deploy anything on conda. Every day I find another case where I reach the infamous:

Cannot remove entries from nonexistent file ~/anaconda3/lib/python3.5/site-packages/easy-install.pth

Here is a very simple way to replicate the problem

conda install flake8
# will install flake 2.6.x with mccabe 0.3.1
python3 -m pip install --upgrade flake8 
# will attempt to install flake8 which needs a new version of mccabe ... and pip will fail to uninstall it

[chebee7i]

An update on this would be nice. I just hit this as well.

[achaiah]

+1 on this. Lots of people are seeing this issue. It can't be that hard to fix. The bug is especially bad for automatic installations where you depend on a requirements file to install dependencies and get kicked out due to this error.

[kalefranz]

I'm just reviewing this issue at the request of @csoja.

@ilanschnell, can we add a post-link script to the conda setuptools package that just touches site-packages/easy-install.pth? From my understanding of the issue, I think that would fix it. And if so, seems like a reasonable thing to do.

[ilanschnell]

@kalefranz why a post-link script? Why not simply add an empty easy-install.pth to the setuptools package instead?

[kalefranz]

Because it would clobber--or attempt to clobber--an otherwise valid easy-install.pth file. We just need to make sure it's there to fix this bug (if my understanding is correct).

[ilanschnell]

Ah, I see. What is the touch command on Windows? Maybe we should just do this on Unix first.

[kalefranz]

I think python setup.py develop uses easy-install.pth. We'd break a lot of stuff for people in non-obvious ways if we just added an empty easy-install.pth file to the setuptools package.

The downside is that it's an unmanaged file that we're creating. So if, for example, an env is switched from python2 to python3, and conda is the only package manager that has ever acted on the env, then lib/python2.7/ will hang around because it's not empty.

[kalefranz]

What is the touch command on Windows?

http://stackoverflow.com/questions/30011267/windows-equivalent-of-touch-i-e-the-node-js-way-to-create-an-index-html

[stxlvt]

Thank you for looking at this bug again. :-)

What is the touch command on Windows?

I tested it, type NUL>>filename is a non-destructive touch equivalent, while type NUL>filename overwrites the content. The former does not update the last modified date if the file already exists, probably because the file has not been modified or created in that case.

[ilanschnell]

If we add the post-link script which touches easy-install.pth, then it will be untracked, and as a result all packages build with conda-build will contain an empty easy-install.pth file. This would create many more problems.

I suggest, we do nothing, and consider this an upstream pip/setuptools problem. If someone likes to step up, we could patch the problem upstream, and apply a patch in the meantime.

[kalefranz]

The former does not update the last modified date if the file already exists, probably because the file has not been modified or created in that case.

Not changing modified date isn't a problem I don't think. The patch is just to make sure the file exists.

[kalefranz]

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

The bug in pip should be patched. And we should either apply that patch to our setuptools build or hack around it with the easy-install.pth fix. We can add a post-unlink script that removes easy-install.pth if it's empty.

[ilanschnell]

The patch to pip could be simply to create an empty easy-install.pth if the files does not exist. This would keep the conda packages clean of hacks.

[ilanschnell]

I'm going ahead and writing a little patch for pip which creates an empty easy-install.pth if it does not exist yet.

[csoja]

Thank you @ilanschnell !

[ilanschnell]

I've added a patch (https://github.com/ContinuumIO/anaconda-recipes/blob/master/pip/ensure_easy_pth.patch) to pip, and build new packages (pip-9.0.1 build 1). However, while this patch is certainly an improvement, pip install -U setuptools still does not work for me because of a different error.

[dstufft]

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

It's not entirely clear to me that this is the case. As best I can tell pip will only care about an easy-install.pth file in the case that it finds either a .egg in site-packages or it's attempting to uninstall something that had been installed in develop mode. In both of these cases I believe setuptools will always ensure that there is a easy-install.pth file. Do you know which case you're hitting to trigger pip to care about this file at all? .egg or develop install?

[ghost]

Trying to install Kivy (Python framework) on Windows 10 from the VS Command prompt (as admin). Getting the same error:

 Cannot remove entries from nonexistent file c:\users\m\anaconda3\lib\site-packages\easy-install.pth

I see that ilanschnell created a patch for this problem, but I am unsure which directory I should place this patch in.

Can someone please advise.

[fmelectrical]

Sorry I used Pythonistribution but after some days it did not work so I uninstalled it and when I wanted to reinstall it I amreceiving this message Doc1.pdf Doc1.pdf

And I cannot install anaconda again

[fmelectrical]

please help me to solve my problem and instal anaconda again and use it

[den-run-ai]

@fmelectrical your issue is not related to this thread at all.

[niklasholm]

@ilanschnell I get this error instead with the latest build of pip:

OSError: [Errno 2] No such file or directory: '[...]/.local/miniconda3/envs/issue542test/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg'

Is it the same you got?

[myidealab]

@niklasholm To resolve the error you got, I reinstalled setuptools and the error went away for the package (newspaper3k) I was trying to import.

    MBPRO-D12LK3WJJC45:~ myidealab$ conda install -c anaconda setuptools=27.2.0
    Fetching package metadata ...........
    Solving package specifications: .

    Package plan for installation in environment /Users/myidealab/anaconda:

    The following packages will be SUPERSEDED by a higher-priority channel:

    conda:      4.3.16-py36_0 --> 4.3.16-py36_0 anaconda
    conda-env:  2.6.0-0       --> 2.6.0-0       anaconda
    setuptools: 27.2.0-py36_0 --> 27.2.0-py36_0 anaconda

    Proceed ([y]/n)? y

    conda-env-2.6. 100% |#################################| Time: 0:00:00 187.75 kB/s
    setuptools-27. 100% |#################################| Time: 0:00:00 919.02 kB/s
    conda-4.3.16-p 100% |#################################| Time: 0:00:00 754.75 kB/s

[ronzillia]

this can be resolved if you do conda install setuptools I am using linux