ModuleNotFoundError: No module named 'pip._vendor.six' when running "pipx install"

[kwbr]

Describe the bug

I have installed a new Debian (unstable) system. On one of my installations pipx runs without problems. The newly installed system behaves differently and I do not know where to look.

How to reproduce

$ pipx install --verbose youtube-dl
pipx > (_package_name_from_spec:93): Determined package name: youtube-dl
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /home/kai/.local/pipx/venvs/youtube-dl
pipx > (run_subprocess:112): running /home/kai/.local/pipx/venvs/youtube-dl/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/kai/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/kai/.local/pipx/venvs/youtube-dl/bin/python --version
pipx > (run_subprocess:112): running /home/kai/.local/pipx/venvs/youtube-dl/bin/python -m pip install youtube-dl
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal.cli.main import main as _main  # isort:skip # noqa
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
ModuleNotFoundError: No module named 'pip._vendor.six'
pipx > (install_package:189): '/home/kai/.local/pipx/venvs/youtube-dl/bin/python -m pip install youtube-dl' failed

pipx > (rmdir:18): removing directory /home/kai/.local/pipx/venvs/youtube-dl
Error installing youtube-dl.

If I run python3 from the command line I can import the module:

$ python3 -c "import pip._vendor.six"
$

I have no idea what could be different. I already checked the list of python3-* packages on both systems but that seems to be the same. Any idea what to look for or how to debug the issue?

Expected behavior

The package should be installed.

[itsayellow]

This looks like a problem with pip to me. For some reason pip is looking for a submodule that is not installed.

  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
ModuleNotFoundError: No module named 'pip._vendor.six'

I don't think this is a pipx error, but looks to me like a problem with the pip installation on that system.

[uranusjr]

@itsayellow How does pipx finds the pip it wants to use? The pip erroring here in from /home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip, which suggests me pipx “controls” that pip installation.

[itsayellow]

@itsayellow How does pipx finds the pip it wants to use? The pip erroring here in from /home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip, which suggests me pipx “controls” that pip installation.

Right, good point.

Well we basically do a pip install pip with some extra switches, using the python that gets installed into the venv for the shared_libs directory. The python installed in the shared_libs directory is what we think is the system python (sys.executable)

[uranusjr]

I see, thanks. WHat would be the best way to repair the pip installation in this case? Would pipx re-build the shared lib directory if we delete it?

[ivanov]

I had the same issue to report here, and tracked it down to the fact that venv is passed the --without-pip flag. If I take that flag away (which is how the previous version of pipx that's currently in Debian - 0.12.3.1- used to work) - then pipx and pip are all happy.

This is because Debian patches pip and

$ python3 -c "import pip._vendor as v; print(v.DEBUNDLED)"
True

On import of pip._vendor, a bunch of the wheel contents for these debundled dependencies get added to sys.path. $ python3 -c "import sys; print(sys.path); import pip._vendor; print(sys.path)"

['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/pi/.local/lib/python3.8/site-packages', '/home/pi/code/pipx2/src', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages'] ['/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl', '/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl', '/usr/share/python-wheels/setuptools-44.0.0-py2.py3-none-any.whl', '/usr/share/python-wheels/pyparsing-2.4.6-py2.py3-none-any.whl', '/usr/share/python-wheels/progress-1.5-py2.py3-none-any.whl', '/usr/share/python-wheels/packaging-20.3-py2.py3-none-any.whl', '/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl', '/usr/share/python-wheels/pip-20.0.2-py2.py3-none-any.whl', '/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl', '/usr/share/python-wheels/lockfile-0.12.2-py2.py3-none-any.whl', '/usr/share/python-wheels/webencodings-0.5.1-py2.py3-none-any.whl', '/usr/share/python-wheels/contextlib2-0.6.0-py2.py3-none-any.whl', '/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl', '/usr/share/python-wheels/distlib-0.3.0-py2.py3-none-any.whl', '/usr/share/python-wheels/six-1.14.0-py2.py3-none-any.whl', '/usr/share/python-wheels/colorama-0.4.3-py2.py3-none-any.whl', '/usr/share/python-wheels/chardet-3.0.4-py2.py3-none-any.whl', '/usr/share/python-wheels/html5lib-1.0.1-py2.py3-none-any.whl', '/usr/share/python-wheels/msgpack-0.6.2-py2.py3-none-any.whl', '/usr/share/python-wheels/wheel-0.34.2-py2.py3-none-any.whl', '/usr/share/python-wheels/certifi-2019.11.28-py2.py3-none-any.whl', '/usr/share/python-wheels/pytoml-0.1.21-py2.py3-none-any.whl', '/usr/share/python-wheels/pep517-0.8.2-py2.py3-none-any.whl', '/usr/share/python-wheels/appdirs-1.4.3-py2.py3-none-any.whl', '/usr/share/python-wheels/requests-2.23.0-py2.py3-none-any.whl', '/usr/share/python-wheels/idna-2.8-py2.py3-none-any.whl', '', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/pi/.local/lib/python3.8/site-packages', '/home/pi/code/pipx2/src', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']

[itsayellow]

We use --without-pip in each user-installed venv because we use a *.pth file that references a shared venv dir just for the purposes of pip, setuptools, wheel.

So the question is, why is the original shared_lib venv not getting installed correctly. And why does installing pip all over again in the tool venv make things work?

[cs01]

WHat would be the best way to repair the pip installation in this case? Would pipx re-build the shared lib directory if we delete it?

Yes, you can erase the dir and pipx will recreate it.

https://github.com/pipxproject/pipx/blob/master/src/pipx/shared_libs.py#L38 https://github.com/pipxproject/pipx/blob/master/src/pipx/shared_libs.py#L32

[ivanov]

So the shared env for me does have the python wheels in ~/.local/pipx/shared/share/python-wheels/ - but the patched pip._vendor in Debian is expecting to find them in a subfolder of the sys.prefix - which ends up being /home/pi/.local/pipx/venvs/cowsay not /home/pi/.local/pipx/venvs/shared

- WHEEL_DIR = os.path.abspath(os.path.dirname(__file__))
+ WHEEL_DIR = os.path.abspath(os.path.join(sys.prefix, 'share', 'python-wheels'))

[ivanov]

(sorry if you're following via email - I made edits to both my earlier posts, I'm pretty sure GH doesn't send updates in these cases)

[ivanov]

To recap - if I activate the shared environment -

20:59@shared$ source bin/activate
(shared) 20:59@shared$ python
Python 3.8.2 (default, Apr  1 2020, 15:52:55) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import sys
>>> sys.path
['', '/usr/lib/python38.zip', '/usr/lib/python3.8',
'/usr/lib/python3.8/lib-dynload',
'/home/pi/.local/pipx/shared/lib/python3.8/site-packages']
>>> import pip._vendor
>>> sys.path
['/home/pi/.local/pipx/shared/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/setuptools-44.0.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/pyparsing-2.4.6-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/progress-1.5-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/packaging-20.3-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/pip-20.0.2-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/lockfile-0.12.2-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/webencodings-0.5.1-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/contextlib2-0.6.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/distlib-0.3.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/six-1.14.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/colorama-0.4.3-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/chardet-3.0.4-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/html5lib-1.0.1-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/msgpack-0.6.2-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/wheel-0.34.2-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/certifi-2019.11.28-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/pytoml-0.1.21-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/pep517-0.8.2-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/appdirs-1.4.3-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/requests-2.23.0-py2.py3-none-any.whl',
'/home/pi/.local/pipx/shared/share/python-wheels/idna-2.8-py2.py3-none-any.whl',
'', '/usr/lib/python38.zip', '/usr/lib/python3.8',
'/usr/lib/python3.8/lib-dynload',
'/home/pi/.local/pipx/shared/lib/python3.8/site-packages']

whereas pipx_pth.write_text in https://github.com/pipxproject/pipx/blob/master/src/pipx/venv.py#L138 assumes that just adding the /home/pi/.local/pipx/shared/lib/python3.8/site-packages line will be sufficient, and this is not the case. If I add all of those shared wheel paths to that .pth file, then again, pipx and pip will be happy.

I'm not sure what the cleanest way of doing this is, though. Perhaps taking the length of sys.path before and after import of pip._vendor, and then take the first entries in sys.path of whatever the difference between those two is, and writing that to the .pth file?

[uranusjr]

I’m inclined to just install a separate copy of pip

[ivanov]

I’m inclined to just install a separate copy of pip

I started this in #389 - and saw that a separate pip makes the cowsay environment 2.4 Mb - whereas the approach in #388 keeps it at 156 Kb.

Let me know what ya'll think :)

[gaborbernat]

This is a Debian bug and should be handled by them, see https://github.com/pre-commit/pre-commit/issues/1383#issuecomment-611091348 for more details. Debian tried to take un-vendoring to next level. We should not make install slower for everyone just to workaround Debians bad patch. My best advice at the moment is either stop using Debian or put pressure on Debian maintainers to stop doing such extreme un-vendoring operations.

[Sispheor]

Same on Ubuntu 20.04... So we cannot use pipx until Debian team update something? With Debian it can take a while, I'm not even sure my grand childrens will see it.

[uranusjr]

Generally I recommend people to leave Debian’s Python alone, and install a separate Python to do their work, e.g. pyenv. This would save you endless headaches if you want to stick with Debian for the long run.

[cs01]

Generally I recommend people to leave Debian’s Python alone, and install a separate Python to do their work, e.g. pyenv. This would save you endless headaches if you want to stick with Debian for the long run.

Thank you for the suggestion, that's a good solution for anyone who is blocked at the moment. However one of the reasons I really liked pipsi and decided to pursue pipx was the idea that you could install things with as little hassle as possible. I was trying to ship an app with Python and my user instructions always felt so ridiculous (create a venv, install it there...). Even if one is not necessarily a Python developer they should be able to take advantage of the ecosystem. It's unfortunate that there is still no good story around this, and requiring casual users who want to use a Python "app" to mess with pyenv is still too high a bar, IMO.

Given the large userbase of Debian/Ubuntu (I am one of them but I guess I haven't been bit by this because I haven't upgraded), this will impact a lot of people, so I think it's worth it to build support into pipx to work around whatever they are doing.

[cs01]

I am an nvm user (node's pyenv), and it's really easy to use. On the other hand, I have looked at pyenv instructions several times over the years, and readme always scares me away. I just go to python.org and install from there and things always work fine.

[Sispheor]

Thanks for the advise. I've installed python rom the source code on my ubuntu 20.04.

I have updated with sudo update-alternatives --config python3

python3 --version

Then I installed pipx with sudo pip3 install pipx

And now when I try to install something

pipx install ansible  
/home/nico/.local/pipx/venvs/ansible/bin/python: No module named pip

Do you have an idea what is missing here? Thanks for your help.

[cs01]

Can you share the output of pipx install ansible --verbose?

[Sispheor]

hre you go

pipx install ansible --verbose 
pipx > (_package_name_from_spec:93): Determined package name: ansible
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /home/nico/.local/pipx/venvs/ansible
pipx > (run_subprocess:112): running /home/nico/.local/pipx/venvs/ansible/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/nico/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/nico/.local/pipx/venvs/ansible/bin/python --version
pipx > (run_subprocess:112): running /home/nico/.local/pipx/venvs/ansible/bin/python -m pip install ansible
/home/nico/.local/pipx/venvs/ansible/bin/python: No module named pip
pipx > (install_package:189): '/home/nico/.local/pipx/venvs/ansible/bin/python -m pip install ansible' failed

pipx > (rmdir:18): removing directory /home/nico/.local/pipx/venvs/ansible
Error installing ansible.

[Sispheor]

pip3 --version
pip 20.0.2 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

[yajo]

How about adding six as a dependency for pipx and mocking pip._vendor.six before calling pip?

[gaborbernat]

Dunno; could we fix this on our side? I think so. Should we? I don't think so. IMHO Debian does here something unreasonable and they should change it to a more sane approach. Would we patch this it would mean we'd start encouraging Debian to do whatever they want and downstream packages will then maintain awkward patches increasing their maintenance burden. I get this is annoying, but please make your voice heard on the Debian issue trackers.

PS. I'm just one of the maintainers here if other maintainers want to accept a patch solving this I'm ok with that. However, my sole opinion is that here Debian should contain the "fix", not the core package.

[uranusjr]

(Shameless plug)

I have looked at pyenv instructions several times over the years, and readme always scares me away. I just go to python.org and install from there and things always work fine.

I have a tool that strips with shell script shim thing, which hopefully makes it more approachable 🙂

https://github.com/uranusjr/pythonup-posix

[Sispheor]

Unfortunately, switching to another python as default breaks everything on my Ubuntu 20.04 system. Like the update manager et other stuff. So i suppose I'll stick to the good old pyenv. Thanks anyway for your support guys. All the best for your module.

[mgedmin]

Workaround for poor new Ubuntu 20.04 LTS users like little old me:

$ ~/.local/pipx/shared/bin/pip install -I pip

This installs the upstream version of pip 20.0.2 in the shared venv, instead of the patched Debian version of pip 20.0.2.

I imagine when upstream pip releases 20.0.3, this problem will go away on its own, because I expect pipx will keep pip upgraded, while Debian won't.

[ivanov]

Just reported this as a bug in the Debian python3-pip package (#958764).

[Sispheor]

You think it's related to python3-pip? Because in my side I've installed pip with the get-pip script to be independent of the distro.

[ivanov]

Examine the pip in your pipx shared environment. If it looks like this:

$ grep -B4  ^WHEEL_DIR ~/.local/pipx/shared/lib/python3.8/site-packages/pip/_vendor/__init__.py

# wish to create their own Wheels for our dependencies to aid in debundling.
prefix = sys.prefix
if sys.prefix.startswith('/usr/lib/pypy'):
    prefix = '/usr'
WHEEL_DIR = os.path.abspath(os.path.join(prefix, 'share', 'python-wheels'))

Then that's the one from Debian. The relevant code in the wheel for pip on PyPI looks like this:

grep -B4  ^WHEEL_DIR pip/_vendor/__init__.py

# By default, look in this directory for a bunch of .whl files which we will
# add to the beginning of sys.path before attempting to import anything. This
# is done to support downstream re-distributors like Debian and Fedora who
# wish to create their own Wheels for our dependencies to aid in debundling.
WHEEL_DIR = os.path.abspath(os.path.dirname(__file__))

[Sispheor]

It means that python3-pip was installed by default. Doesn't it? So I would just have to delete the system package before installing it via get-pip and it would be ok?

[ivanov]

For me, removing python3-pip didn't fix it. Looks like the pip that pipx is picking up for the shared environment is coming from python-pip-whl package which is necessary for python3-venv which pipx needs. Can you try #388 and report the result on that PR?

[ivanov]

Just an update that this issue (filed as Debian Bug #958764) has now been fixed by in the latest python3-pip package in Debian Unstable (Version: 20.1-1). The patch of interest for Debian derivatives entails switching to using sys.base_prefix in place of sys.prefix for the unbundling logic, see this commit.

[nhumrich]

Just want to add that I had this problem when installed python3.9, migrating form 3.8. (not on debian) Installing 3.9 made my existing virtual envs not work, and I got this error. Completely removing ~/.local/pipx, then reinstalling my pipx things solved the problem.

[monkut]

Same issue here when using pyenv and pipenv:

[pipenv.exceptions.InstallError]: from pip._vendor.six import iteritems [pipenv.exceptions.InstallError]: ModuleNotFoundError: No module named 'pip._vendor.six' ERROR: Couldn't install package: certifi

When preparing to use pyenv, I mistakenly included python3-dev during apt install. This apparently installs a local version of pip which leads to a conflict of some kind when running pipenv install ....

I resolved the issue by removing python3-dev and installing the necessary packages separately:

# remove python3-dev
sudo apt remove python3-dev

# re-add packages needed for compiling python
# from:
# https://devguide.python.org/setup/
sudo apt-get install build-essential gdb lcov libbz2-dev libffi-dev \
      libgdbm-dev liblzma-dev libncurses5-dev libreadline6-dev \
      libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev

After cleaning this up I was able to run pipenv install .. as expected.