--sync removes pkg-resources, leading to pillow error

[ClaudiaSchulz]

• Poetry version: 1.2.1
• Python version: 3.8.10
• OS version and name: Ubuntu 20.04.3 LTS
• pyproject.toml:

  
  [tool.poetry]
  name = "test1"
  version = "0.1.0"
  description = ""
  authors = ["Claudia"]

[tool.poetry.dependencies] python = "~3.8"

[tool.poetry.dev-dependencies]

[build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"

## Issue
When running `poetry install --sync`, poetry removes anything pre-installed (as it should), unfortunately this includes `pkg-resources` (see #2673 for a discussion), which is pre-installed.
Removing this package leads to an error when trying to then `poetry add pillow`, as discussed in #5104.

I also tried this with Poetry 1.2.0b2, mentioned in #5104 - same problem.

[neersighted]

There's not enough detail here to reproduce this (where is a preinstalled pkg_resources coming from?) -- here's what I tried:

docker run --rm -i --entrypoint /bin/bash python:3.8 <<EOF
curl https://install.python-poetry.org | POETRY_HOME=/opt/poetry python3 - --version 1.2.1
export PATH="/opt/poetry/bin:$PATH"
poetry new foobar
pushd foobar
poetry lock
poetry install --sync
poetry add pillow
EOF

[dimbleby]

https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463 maybe

[neersighted]

Tried again with Ubuntu 20.04:

docker run --rm -i --entrypoint /bin/bash ubuntu:20.04 <<EOF
apt-get update
apt-get install -y python3 python3-venv python3-pip curl
curl https://install.python-poetry.org | POETRY_HOME=/opt/poetry python3 - --version 1.2.1
export PATH="/opt/poetry/bin:$PATH"
poetry new foobar
pushd foobar
poetry lock
poetry install --sync
poetry add pillow
EOF

[neersighted]

I would check you're really on the Poetry version you think you're on -- the ~3.8 specifier in the example pyproject.toml looks a bit suspicious since we write ^X.Y by default these days.

[dimbleby]

$ python3 -m venv ~/.virtualenvs/foo
$ ls ~/.virtualenvs/foo/lib/python3.8/site-packages/
__pycache__  easy_install.py  pip  pip-20.0.2.dist-info  pkg_resources  pkg_resources-0.0.0.dist-info  setuptools  setuptools-44.0.0.dist-info

that pkg_resources presumably.

this is with ubuntu 20.04 and the system python. If I use a deadsnakes python3.9, then it's not there:

$ ls ~/.virtualenvs/foo/lib/python3.9/site-packages/
_distutils_hack  distutils-precedence.pth  pip  pip-22.0.4.dist-info  pkg_resources  setuptools  setuptools-58.1.0.dist-info

I don't intend to dig further, my money is on it being connected to or the same as that ubuntu bug I linked yesterday

[neersighted]

Interesting -- that does look like Ubuntu is doing some extremely sketchy stuff and it's not Poetry's fault...

[dimbleby]

I wondered why poetry didn't also uninstall pip and setuptools in that case, the answer is that there's some special-case code.

So if you wanted to treat pkg-resources as a special case to cope with this, I guess that's where to do it.

[neersighted]

Right, but uninstalling that nonsense pkg_resources shouldn't break pip... I guess what might be happening is the distro-provided pip copied from the distro's site-packages is debundled and depends on that pkg_resources... If the pip was upgraded then this wouldn't be an issue (you'd get a properly vendored version from PyPI), but since it isn't, we break the pip in the virtualenv?

This really seems like a Ubuntu issue to me, in that case. This would be solved both by #6458 or Poetry using its own installer code (#6205).

[dimbleby]

it shouldn't only be about pip and installation though.

If a project is using pkg_resources and is in an environment where pkg_resources is provided by this strange debundled version - then uninstalling that package will break the project.

[neersighted]

I was assuming that if the project used pkg_resources, it would declare it, Poetry would upgrade the nonsense version to the locked version, and pip would probably not break -- but fair point, there could be more sharp edges here.

[dimbleby]

but you wouldn't declare an explicit dependency on pkg_resources, you'd declare a dependency on setuptools.

So then poetry would see no reason to keep the unbundled pkg_resources around

[neersighted]

That is a fair point -- maybe we just hold our nose and whitelist pkg_resources with a link back here explaining why...

[ClaudiaSchulz]

@neersighted that sounds like a good solution, even though I agree that the pkg_resources is somewhat sketchy (unfortunately I have no control over my container environment, so there's no way for me to use a different OS version). But it sounds like there wouldn't be any harm in whitelisting pkg-resources in the same way that setuptools is whitelisted.

[ClaudiaSchulz]

Also, adding pgk_resources as an explicit dependency doesn't work since Poetry is unable to find the package.

[espdev]

Poetry 1.2.2 removes setuptools from my project:

$ poetry install

Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing setuptools (65.5.0)

And google APIs do not work without pkg_resources. I have added setuptools as dependency to the project:

$ poetry add setuptools 

It seems it works but looks very strange.

[neersighted]

This is presumably because setuptools is required by an optional dependency you are not installing, so Poetry considers it a managed package. You do explicitly depend on pkg_resources which is shipped with setuptools, so it is correct to declare a dependency.

It's hard to provide more insight without seeing your pyproject.toml -- I suggest asking in Discord or on Discussions if you have more questions.

[macgeneral]

I run into this problem for Docker images based on Ubuntu 20.04 LTS as well.

I know it's an ugly workaround, but running

python3 -m pip install --no-cache-dir --force-reinstall --upgrade setuptools

after poetry [..] --sync [..] seems to fix the otherwise broken virtual environment.