search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.53k stars 3.03k forks source link

ImportError in system pip wrappers after an upgrade #5599

Closed pfmoore closed 3 years ago

pfmoore commented 6 years ago

Overview

After upgrading to pip 10 or higher, many users are encountering error like

These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we'll try to help you solve your issue, this is not the "fault" of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

You can reach out to your Python provider (eg: Linux Distro, Cloud Provider, etc) for help with this issue.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

  1. Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
  2. Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
  3. Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

Debugging the Issue

Before trying to work out what's going on, it's critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That's often in the traceback you get, but if it isn't, you can use OS tools like which, or Python's shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can't, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you're usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that's why it imports pip.main) under a Python interpreter that sees a copy of pip that's version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that's where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution's pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it's quite likely with problems of this nature that you'll need to modify system-supplied files or settings. You do so at your own risk. If you're not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There's only so much that can be achieved remotely.

Also, the pip developers don't provide any guarantees that advice in the comments on this issue is correct, or that it won't damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue:

5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

denisglotov commented 6 years ago

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side. To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

randomdudethathelpsyou commented 6 years ago

I am pretty new to coding, however I managed to fix it in a kind of odd way. My problem was that there was nothing in my /usr/bin/ folder regarding pip. to fix this I had my buddy upload the get-pip.py program to raspberry pi, from which I downloaded the code to my computer and ran it. It then over-rode the old pip program and installed the new one.

Here's the code we used: working computer to pi: scp /whatever_your_directory_is/get-pip.py pi@pi's_address:/pi's_directory pi to my computer (the last period is important): scp pi@10.200.130.65:/pi's_directoryi/get-pip.py . to download pip: sudo python get-pip.py

Sorry if it's weird, but that's what got it working for me

veneman commented 6 years ago

After upgrading pip I ran into the error ImportError: cannot import name 'py31compat' when trying to import pyarrow. I have a conda installation and after rolling back pip to version 9 on all of my environments I still ran into the error.

The fix for me was to change line 72 in site-packages/pkg_resources/__init__.py from from . import py31compat to from tempfile import TemporaryDirectory.

setuptools/py31compat.py is a workaround for older versions of Python that don't have TemporaryDirectory. Since I'm running Python 3.6 this change eliminates the need to use py31compat.

Things seem to be working after this change but I'm not that confident it'll keep working down the road.

ErichBSchulz commented 6 years ago

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit :-/

pfmoore commented 6 years ago

@ErichBSchulz We're aware of that - see #5346.

Maybe I should have more explicitly said "Only ever use your system package manager to upgrade your system pip". The more detailed comment in that bullet point makes it clearer, but I've edited the post to make it explicit.

MannyC commented 6 years ago

I followed some instructions elsewhere and ran 

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue. 

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

PieterGit commented 6 years ago

I would add this advice of not upgrading your system pip in the documentation. I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page: https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:

python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip

After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip The request to update pip in debian repo can be found here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

borismo commented 6 years ago

pip3 install --user pip seems to break my system pip.

Isn't it a good idea to install the latest version of pip in the user's home directory?

abhiTronix commented 6 years ago

The Ubuntu's python-pip-9.0.1 installed(via apt get python-pip) is conflicting with Latest pip-18.0 provided by pypa. So sudo rm -rf ~/.local/lib/python2.7/site-packages/(Beware!) removes the ubuntu's python-pip and resolves the problem. Only one pip can exist on the system. Thanks @denisglotov

Also Upgrading pip doesn't help as it does'nt overwrite the preexisting installed python-pip but instead create another copy of pip in different directory causing the conflict.

Ark-kun commented 6 years ago

My problem was installing pip in ~/.local/ and having at the end of $PATH.

ddombrowsky commented 6 years ago

I wonder how npm and cpan solve this problem? In any case, I just uninstalled system pip3 and created my own:

#!/bin/sh

exec python3 -m pip "$@"

I'm the only user running it on this system anyway. Victory!

axodd commented 6 years ago

Related discussion (from April) with a variety of solutions from Stackoverflow community: https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main https://stackoverflow.com/questions/49964093 Let's hope #5346 will help minimize future impacts, as OS distributions alone may be too slow at preventing this issue from happening in their LTS and later releases..

hawran commented 6 years ago

...

Debugging the Issue

... Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term). ...

Well:

$ python -m pip
/usr/bin/python: No module named pip
$ python3 -m pip3
/usr/bin/python3: No module named pip3

OK, so let's try something else:

$ python3 -m pip uninstall pip3
Skipping pip3 as it is not installed.
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Wait, 'You are using pip version 10.0.1, however version 18.1 is available.'

What?

$ dpkg -l '*pip*'
...                                                                  Version                                   Architecture                              Description
+++-=========================================================================-=========================================-=========================================-=======================================================================================================================================================
ii  libpipeline1:amd64                                                        1.4.1-2                                   amd64                                     pipeline manipulation library
un  pppdcapiplugin                                                            <none>                                    <none>                                    (no description available)
$

So what am I supposed to do now?

PS I really consider the sentence: 'You are using pip version 10.0.1, however version 18.1 is available.' really, really confusing...

hawran commented 6 years ago

...

Update: Finally I've managed to fix my system using the https://github.com/pypa/pip/issues/5599#issuecomment-416318017. Thank you very much!

Xubuntu

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial
johncoleman83 commented 5 years ago

I just ran into this problem. I believe my problem was because of the referenced use of --user since the python shebang path in the pip wrapper was set to: #!/usr/local/opt/python3/bin/python3.6, yet all my python packages and python installs are in /usr/local/bin with sym links to homebrew directories /usr/local/Cellar/.... I was going to edit the pip paths, but decided to preference an uninstall & new installation of pip with these steps:

I was going to uninstall python3 with brew, but since I needed to also update my python from 3.6.3 to 3.7.1, I just skipped the removal of older python3 versions.

Labhras commented 5 years ago

I was installing devstack (on CentOS on VirtualBox) when I hit this error. I have a root user and a devstack user. devstack user pip version was 18.1 [stack@DevStack devstack]$ python -m pip --version pip 18.1 from /opt/stack/.local/lib/python2.7/site-packages/pip (python 2.7) but stack install was downgrading root users pip version to 9.x (really annoying!) [root@DevStack ~]# python -m pip --version pip 9.0.3 from /usr/lib/python2.7/site-packages (python 2.7)

I noticed in the devstack install: Collecting pip!=8,<10 (from -c /opt/stack/repos/devstack/tools/cap-pip.txt (line 1)) So changed cap-pip.txt to pip!=17,<19

Not sure why devstack are enforcing a version

lxlee commented 5 years ago

I ran into the same problem. It seems that there is a conflict between pip in system and pip in python site-packages for me in my ubuntu. After upgrading the pip as the pip command suggests, I removed the system pip using sudo apt remove python-pip And restarting the bash, the error goes away.

ghost commented 5 years ago

@pfmoore

  1. In this comment you said "using python -m pip it's still not supported". The documentation instead says to call pip from another script like this:

subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package'])

which of course will ultimately call python -m pip. So who is right?

  1. You also said that "from pip import main was never supported". So what is the correct way to call pip? The documentation says to call

$ pip <pip arguments>

but that of course will call the unsupported distribution-specific wrapper, which, in the case of Ubuntu, uses the unsupported .from pip import main

ghost commented 5 years ago

@pfmoore

  1. I have installed pip in a virtualenv using the get-pip.py script, a method described in the documentation, but in a virtual environment as not to conflict with the distribution's pip:
virtualenv -p python3 --no-pip pipinstall
source ./pipinstall/bin/activate
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py
cat ./pipinstall/bin/pip3.6

Here is the content of the file ./pipinstall/bin/pip3.6

#!/home/raffaele/bin/pyvenv/pipinstall/bin/python3

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

The unsupported from pip._internal import main is being used instead of subprocess.call().

Whose bug is it?

pfmoore commented 5 years ago

pip3.6 is a script installed by (and part of) pip itself. So it is allowed to use pip internals.

Ubuntu supply their own pip wrapper script, which means they take on the support burden of using (and supporting/maintaining) calls to undocumented internal functions.

ivarec commented 5 years ago

I ran into the same problem. It seems that there is a conflict between pip in system and pip in python site-packages for me in my ubuntu. After upgrading the pip as the pip command suggests, I removed the system pip using sudo apt remove python-pip And restarting the bash, the error goes away.

This solution worked for me. Restarting bash was what I was missing before. Messy!

Tahasd commented 5 years ago

I have a more common request but related to that issue. I tried uprgrading pip in the not adviced way and run into that main error problem.

I will repair my system python according to the advices (thx very much for the detail description, helped a lot) but now I have a question related to this. Sorry in advance, I would not consider myself very experienced, so I dont know if this is the wrong issue to post this or if the question is redundant. But still it is a question to circumvent the problem caused by upgrading the system pip.

So my question is: is it possible to install a python version next to the system python and use this as default under the following conditions:

Related to that:

I tried to search the internet for these questions and found mostly things related to envs and mutliple python installations with different versions but not the same. I also stumbled upon pipenv but this seems again to be just the same as using envs.

Regarding the --user statement, if I got it right can be used to install a newer version of pip for the current user. Then I have to adjust the paths of the default pip to that one, right? And does it require already a sort of alternative python version installed for the current user?

Something like installing an alternative python for a user is probably what I am looking for and make the user`s python the default python for that user. But I did not find issues or topics concretely addressing this.

If someone has an idea about this and can share it with me I would be very thankful. And maybe this helps also others to set up a default python that can be altered in any desired way without interfering with the system python.

hodlon commented 5 years ago

I ran into this same issue and was able to solve with this advice:

https://github.com/pypa/pip/issues/5599#issuecomment-420516760

Thanks much @denisglotov and @abhiTronix

dasbuilder commented 5 years ago

One thing that I've run across is that I'm using 3.7.2 which currently isn't included with Ubuntu's default package manager. Also, every time I have tried to upgrade pip using non-sudo, I've always received an error. Since 3.7 isn't officially available (that I'm currently aware of), I built both 3.7.2 and pip3 from source and then ensured that files in my /usr/local/lib/python3.7 directory were all owned by my user and not root.

I guess that's why it's also recommended to use a virtualenv as to not mess up your systems default version?

Akababa commented 5 years ago

I fixed this by opening the files /usr/local/bin/pip and /usr/bin/pip and replacing the line from pip import main with from pip._internal import main.

irishquinn commented 5 years ago

Did you try

$ hash -d pip

Or in dash (sh):

$ hash -r pip
Quiigi commented 5 years ago

Summary: How can i rewrite pip using subprocess?

I'm trying to install the R package keras from RStudio. Under the hood, it uses Python. When I run the R function install_keras(), it hits the infamous ImportError: cannot import name main.

@pfmoore :

So it's all local to my r-tensorflow, installed this week. No upgrade or system pip seems involved.

You give two pieces of advice: (A) clear the hash, or (B) rewrite using subprocess.

In a shell, I do

$ hash -d pip
bash: hash: pip: not found

So how do I rewrite ~/.virtualenvs/r-tensorflow/bin/pip using subprocess instead of importing from pip? Here is the complete script:

#!/home/brech/.virtualenvs/r-tensorflow/bin/python

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
pfmoore commented 5 years ago

@Quiigi That pip script is not the one shipped with pip. You should discuss the issue with whoever or whatever generated that script.

To give you some suggestions:

  1. As it's in a .virtualenvs directory, presumably pip (and that script) was installed when the virtualenv was created. Start looking there - how did the virtualenv get created?
  2. The script looks like the sort of thing easy_install creates - did you (or whatever built the environment) install pip using easy_install? We don't support that, so you should install pip "properly" (via virtualenv, or via get-pip.py)

This actually looks suspiciously like something I've seen before, with PyCharm. Are you using PyCharm? If so there's a known bug in their "build an environment" stuff that installs pip using easy_install, so that may be your issue here. I raised a bug with them about it, but I don't know the bug ID (I stopped caring about PyCharm because they didn't seem interested in fixing this issue :-))

akhanivishant commented 5 years ago

Traceback (most recent call last): File "/usr/bin/pip", line 9, in from pip import main ImportError: cannot import name main..

I am using virtulaenv and there is only one pip , and still getting this error while useing pip.

Quiigi commented 5 years ago

@pfmoore It's from tensorflow, not PyCharm. See install.R; below are the two functions that look most relevant to me. And tensorflow shoots itself in the foot where it calls pip_install three times:

This code finds that the native pip 1.5.4 is older than 8.1, so it tries to upgrade 3 packages. First pip, which works because the pip script that tensorflow installed into my virtualenv, and which I reproduced above, is compatible with 1.5.4, and thus upgrades to pip 19.0.3.

Next (still because pip 1.5 is older than 8.1) it tries to upgrade wheel, but now the same supplied script ~/.virtualenvs/r-tensorflow/bin/pip is incompatible with the just-installed pip 19.0.3 from ~/.virtualenvs/r-tensorflow/local/lib/python2.7/site-packages/pip, which no longer provides main. So the second invocation of pip fails, and thus the whole installation of tensorflow!

  # function to call pip within virtual env
  pip_install <- function(pkgs, message) {
    cmd <- sprintf("%ssource %s && %s install --ignore-installed --upgrade %s%s",
                   ifelse(is_osx(), "", "/bin/bash -c \""),
                   shQuote(path.expand(virtualenv_bin("activate"))),
                   shQuote(path.expand(virtualenv_bin(pip_version))),
                   paste(shQuote(pkgs), collapse = " "),
                   ifelse(is_osx(), "", "\""))
    cat(message, "...\n")
    result <- system(cmd)
    if (result != 0L)
      stop("Error ", result, " occurred installing TensorFlow", call. = FALSE)
  }

  # upgrade pip and related utilities if its older than 8.1
  if (installed_pip_version() < "8.1") {
    pip_install("pip", "Upgrading pip")
    pip_install("wheel", "Upgrading wheel")
    pip_install("setuptools", "Upgrading setuptools")
  }

My workaround was to do the following in a shell:

   $ ~/.virtualenvs/r-tensorflow/bin/python -m pip install \
      --ignore-installed --upgrade tensorflow==1.12.0 h5py pyyaml requests \
      Pillow keras tensorflow-hub tensorflow-probability scipy
pfmoore commented 5 years ago

@pfmoore It's from tensorflow

OK, so you should probably raise this with them. It looks like their install process is writing an incorrect form of the pip wrapper script.

If that script is upgrading the system-supplied pip using pip (rather than using the standard OS packages) then that's not supported, so again that's a problem with the tensorflow supplied install script (which, if it's modifying the system pip, you're presumably running with sudo, which is also not recommended/supported).

Anyhow, the main point is that this appears to be a tensorflow issue, not a pip issue.

xiuliren commented 5 years ago

ln -sf /usr/local/bin/pip3 /usr/bin/pip works.

cetooley commented 5 years ago

So, I had this problem, and apparently it was because i have a local copy installed along with the system version - removing my local version: rm ~/.local/bin/pip3* and then reinstalling from repo sudo apt install --reinstall python3-pip worked for me.

namhong2001 commented 5 years ago

(For ubuntu users) Final solution! No action required! You may want to upgrade pip using python3 -m pip install --user --upgrade pip But error occurs! This because new pip locates in $HOME/.local/bin but your $PATH doesn't point it. But actually .profile have a script to load /.local/bin to path! So just logout and login again. Then problem fixed. By the way, it seems make sense to have pip packages in user level and load it via .profile because this also for user level.

ANorwell commented 5 years ago

It seems to me that the advice: Only ever use your system package manager to upgrade the system pip. isn't really sufficient: commands like pip3 install pipenv will cause pip to be upgraded, stopping it (or rather, the ubuntu 16.04 wrapper) from working.

chrahunt commented 5 years ago

In 19.3 we moved the main function from pip._internal to pip._internal.main, so you may see output like:

Traceback (most recent call last):
  File "/usr/bin/pip3", line 16, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable

The advice in the original post still holds.

jayvdb commented 5 years ago

That new output is now occurring when prior versions succeeded, breaking CI at https://github.com/theacodes/nox/

It looks like

$ pip install wheel
Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python37\Scripts\pip.exe\__main__.py", line 9, in <module>
TypeError: 'module' object is not callable

See https://ci.appveyor.com/project/theacodes/nox/builds/28111820/job/64e3ug9ibm0vkteo

AvnerCohen commented 5 years ago

Sharing, if anyone else coming across this as of 19.3:

$ python -m pip --version
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/commands/__init__.py", line 9, in <module>
    from pip._internal.commands.download import DownloadCommand
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 10, in <module>
    from pip._internal.operations.prepare import RequirementPreparer
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 9, in <module>
    from pip._internal.distributions import (
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/distributions/__init__.py", line 1, in <module>
    from pip._internal.distributions.source import SourceDistribution
ImportError: cannot import name 'SourceDistribution'
AvnerCohen commented 5 years ago

@pfmoore It might not be the best place to ask, but I'll try anyway.

I suspect this repeating issue, of maybe a badly configured /workflow for pip installs is a repeating theme and more specifically highlighted on CI systems.

For the requirements of:

  1. Start a new test (for TravisCI, CircleCI and friends means run a new docker instance)
  2. Install a python version (or use the existing one from the image)
  3. Create a virtual environment.
  4. Upgrade pip to latest (or to explicit version).
  5. Install the specific dependencies of the software being tested.

Maybe there should be a cook-book documented on how to achieve that?

pfmoore commented 5 years ago

@AvnerCohen That sounds like a very good idea (specifically "setting up pip on common CI systems"). It's probably something that could be written as a subsection within the "User Guide" section of the pip documentation. Would you be interested in contributing such a section?

Going beyond CI systems to cover all of the various possibilities for user environments in general is too much, though. We have sections on installing and using pip, and I think it's reasonable to write them based on the assumption that the user knows how to use and manage their computing environment. (That's not actually true in many cases, people often don't understand their environments for perfectly acceptable reasons, but I don't think it's the place of the pip manual to cover that general issue).

AvnerCohen commented 5 years ago

Thanks @pfmoore . Would love to help on that, but I think what I am doing right now is wrong since we keep coming across such issues. so not sure this should be presented as best practice.

I guess it would be best to discuss this over a dedicated PR, I'll stub something, taking into account also the "General Advice" and "Fixing the Issue" from this ticket and hopefully we can create something useful for others as well.

pfmoore commented 5 years ago

OK, cool. I think one of the problems here is that no-one knows what "best practice" is (and everyone's quite reasonably focused on fixing their own problem, not working out a general solution), which is why we keep getting the same sort of questions. So having somewhere we can thrash out the "right" solution will be a good step forward.

vinesmsuic commented 5 years ago

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

You will need to clear the cache since the terminal will give : pip: No such file or directory

$ hash -r
felix-ht commented 5 years ago

@pradyunsg Any plans to fix this? This issue is open since more than a year ago, an certainly quite critical for pip3 and all linux users!

If for whatever reason it cannot be fixed, an "official" statement on how we are supposed to upgrade would be great.

pfmoore commented 5 years ago

@oteph There's nothing to fix here. See the original issue description, specifically:

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip.

We can't provide an "official" statement on how to upgrade scripts (system pip wrappers) that we didn't provide and that we don't maintain.

felix-ht commented 5 years ago

@pfmoore Thanks for the clarification!

Here is how i fixed it for myself:

Assuming you installed the newest version of pip with pip3 install --upgrade pip --user and it is now broken with the dreadful ImportError: cannot import name 'main'

This is how you can get a working installation: 

echo "export PATH=~/.local/bin:$PATH" >> ~/.bashrc
source ~/.bashrc

Now pip3 should now work as expected. Note that this only masks the issue.

If it is not yet broken execute:

pip3 install --upgrade pip --user 
echo "export PATH=~/.local/bin:$PATH" >> ~/.bashrc
source ~/.bashrc
njdjacobs commented 4 years ago

So to use the latest version of pip, we have to install a local (per-user) version of pip, and leave the system-installed pip untouched. This is clear and helpful advice. Thanks. I am very lucky I saw this page in time, before using sudo to install.

But ... what a mess. On a system with N users, there could be N separately-installed versions of pip, some of them identical. Spasmodically, all N users will decide they should have the latest and greatest version of pip, and upgrade their copies asynchronously. Creating this situation was a massive blunder. I'm sure it's extremely difficult to fix now, but it's really important!

uranusjr commented 4 years ago

There is a simple way to “solve” this: Upgrade the system pip with the system package manager. If you can (say) apt upgrade to the latest pip, everyone can use that. If you can’t, the problem is in the system, and per-user installs is the best pip maintainers can offer.

infamybsd commented 4 years ago

I had the same experience with this issue #5373. The only solution for me was uninstalling the pip.

sudo apt remove python3-pip sudo apt purge python3-pip

And follow the installation guide on this site https://pip.pypa.io/en/stable/installing/ Now pip works like a charm.

Sieboldianus commented 4 years ago

I had some related problem in Ubuntu in WSL. I had installed pip with --user and since --upgrade, modules were not found. There weren't many system-level pip packages installed (just docker-compose), so I decided to fix my system pip from scratch using apt-get.

apt-get installs a much older version of pip which can lead to problems, completely remove python-pip with:

apt-get remove --purge python-pip

then:

curl https://bootstrap.pypa.io/get-pip.py | sudo python

using sudo if required