Package upgrade - causes a corrupted env

[simingy]

Environment

• pip version: 19.3.1
• Python version: 3.6.9
• OS: macOS 10.15.1 catalina

inside a virtual environment created using python3 -m venv note that all packages are in .whl format.

Description Given this package: https://pypi.org/project/unicon/ In all versions of Unicon up to and including v19.10, it was a single package.

in v19.11 we split the package into unicon, and unicon.plugins. namely, the folder unicon/plugins was moved from the single "unicon" package, into unicon.plugins package, with Unicon now having unicon.plugins as a dependency.

the idea was that we could just pip install --upgrade unicon and because of the dependency, the plugins package is installed automatically.

This works during clean install, but if anyone had unicon package <19.11, and does an upgrade, most files from unicon/plugins folder are missing.

I think the reasoning behind this bug is that:

• the files under unicon/plugins folder was originally part of unicon <v19.11
• the new unicon v19.10 doesn't have this folder - it was moved to unicon.plugins
• pip thinks, after installation, that the files from this folder is save to remove (because they are described by unicon older versions' pkg info file, and deletes them
• but this really deleted the new package's files.

Expected behavior

1. pip install --upgrade would uninstall the old package
2. install the new package and its dependencies

or

on upgrade, don't delete files from the dependency package.

How to Reproduce

1. create a venv
2. pip install unicon==19.10, check files exists under lib/python*/site-packages/unicon/plugins
3. pip install --upgrade unicon, check files are mostly missing from the above directory

Output

(1) ~/workspace/trial/1 pip install unicon==19.10
Collecting unicon==19.10
  Using cached https://files.pythonhosted.org/packages/e1/d9/56515b489147fd4291207829f3e7ba91466bdc6287075309fbdda7b071a8/unicon-19.10-cp36-cp36m-macosx_10_10_x86_64.whl
Requirement already satisfied: dill in ./lib/python3.6/site-packages (from unicon==19.10) (0.3.1.1)
Requirement already satisfied: pyyaml in ./lib/python3.6/site-packages (from unicon==19.10) (5.1.2)
Requirement already satisfied: setuptools in ./lib/python3.6/site-packages (from unicon==19.10) (42.0.1)
Installing collected packages: unicon
Successfully installed unicon-19.10
(1) ~/workspace/trial/1 ls lib/python3.6/site-packages/unicon/plugins
__init__.py aci         asa         cimc        fxos        ios         iosxr       junos       nso         staros      vos
__pycache__ aireos      cheetah     confd       generic     iosxe       ise         linux       nxos        utils.py
(1) ~/workspace/trial/1 pip install --upgrade unicon
Collecting unicon
  Using cached https://files.pythonhosted.org/packages/5a/e2/ab0a5dbc27bfdd25491e6a06648ed0146ddad1efc844a67c191bdeaeb5d2/unicon-19.11-cp36-cp36m-macosx_10_10_x86_64.whl
Requirement already satisfied, skipping upgrade: pyyaml in ./lib/python3.6/site-packages (from unicon) (5.1.2)
Collecting unicon.plugins<19.12.0,>=19.11.0
  Using cached https://files.pythonhosted.org/packages/80/af/e1ed36b2e4d4bcdaa3f1542d17012d8a9ba919fe6664e6e8c2ed97e8c233/unicon.plugins-19.11-py3-none-any.whl
Requirement already satisfied, skipping upgrade: dill in ./lib/python3.6/site-packages (from unicon) (0.3.1.1)
Installing collected packages: unicon.plugins, unicon
  Found existing installation: unicon 19.10
    Uninstalling unicon-19.10:
      Successfully uninstalled unicon-19.10
Successfully installed unicon-19.11 unicon.plugins-19.11
(1) ~/workspace/trial/1 ls lib/python3.6/site-packages/unicon/plugins
sdwan tests

[simingy]

traced this a bit more, it seems like

1. because unicon.plugins is a dependency to unicon, it gets put in place first (installed)
2. now pip tries to upgrade unicon, which removes the old version first - because the old version uses the path as described by unicon.plugins, it removes what was just installed from step 1

hence the conflict.

should these installs be sandboxed?

[xavfernandez]

Thanks for the report. It is strongly related to #4625.

[pradyunsg]

This is functionally the same issue as https://github.com/pypa/pip/issues/4625 but showing up in the context of an uninstall, and causing a breakage due to install ordering. I'll consolidate this into that one, since resolving that issue will involve figuring out what to do in such cases.