search

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.
http://iree.dev/
Apache License 2.0
2.86k stars 625 forks source link

Strange behavior trying to upgrade iree-base-compiler Python package #19305

Open ScottTodd opened 5 days ago

ScottTodd commented 5 days ago

What happened?

When old iree-base-compiler and iree-base-runtime packages are installed [into a venv] and one attempts to upgrade both, the iree-base-compiler package appears to not be upgraded:

pip install -f https://iree.dev/pip-release-links.html --upgrade --pre \
  iree-base-compiler \
  iree-base-runtime

Requirement already satisfied: iree-base-compiler in /groups/aig_sharks/actions-runner-nod-gpu5/_work/_tool/Python/3.11.9/x64/lib/python3.11/site-packages (3.0.0rc20241114)
Collecting iree-base-compiler
  Downloading https://github.com/iree-org/iree/releases/download/iree-3.1.0rc20241126/iree_base_compiler-3.1.0rc20241126-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (71.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.3/71.3 MB 49.5 MB/s eta 0:00:00
Requirement already satisfied: iree-base-runtime in /groups/aig_sharks/actions-runner-nod-gpu5/_work/_tool/Python/3.11.9/x64/lib/python3.11/site-packages (3.1.0rc20241125)
Collecting iree-base-runtime
  Downloading https://github.com/iree-org/iree/releases/download/iree-3.1.0rc20241126/iree_base_runtime-3.1.0rc20241126-cp311-cp311-manylinux_2_28_x86_64.whl (8.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 MB 51.4 MB/s eta 0:00:00
Installing collected packages: iree-base-runtime, iree-base-compiler
  Attempting uninstall: iree-base-runtime
    Found existing installation: iree-base-runtime 3.1.0rc20241125
    Uninstalling iree-base-runtime-3.1.0rc20241125:
      Successfully uninstalled iree-base-runtime-3.1.0rc20241125
  Attempting uninstall: iree-base-compiler
    Found existing installation: iree-base-compiler 3.0.0rc20241114
    Uninstalling iree-base-compiler-3.0.0rc20241114:
      Successfully uninstalled iree-base-compiler-3.0.0rc20241114
Successfully installed iree-base-compiler-3.0.0rc20241114 iree-base-runtime-3.1.0rc20241126

https://github.com/nod-ai/shark-ai/actions/runs/12021435366/job/33546714200#step:5:203

Steps to reproduce your issue

  1. Create a venv
  2. Install old packages
  3. Try to upgrade to new packages
  4. See that iree-base-compiler is still using an older version

We should test with stable packages, nightly packages, and dev packages. The upgrade path for stable -> stable should work, while upgrading/downgrading with nightly or dev releases can require extra steps (like deleting the venv).

What component(s) does this issue relate to?

Python

Version information

Observed with packages newer than 3.0.0. Should check if 2.9.0 had this issue too, since https://github.com/iree-org/iree/pull/19156 and other packaging-related changes landed more recently.

Additional context

It's possible that https://github.com/iree-org/iree/pull/19156 broke the upgrade path somehow in the process of fixing the version reporting (https://github.com/iree-org/iree/issues/19155).

marbre commented 4 days ago

With

python3 -m venv venv-upgrade-path
source venv-upgrade-path/bin/activate
pip install -f https://iree.dev/pip-release-links.html --pre iree-base-runtime==3.0.0rc20241114 iree-base-compiler==3.0.0rc20241114
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade iree-base-compiler iree-base-runtime

I wasn't able to reproduce. Before the upgrade pip freeze| grep iree gets me

iree-base-compiler==3.0.0rc20241114 iree-base-runtime==3.0.0rc20241114

after upgrading I have

iree-base-compiler==3.1.0rc20241127 iree-base-runtime==3.1.0rc20241127

Tested on Ubuntu 24.04 with Python 3.12.3 (system package) and pip 24.0 (system package).

Have you any other packages installed in your venv that depend on one of the IREE packages (e.g. iree-turbine?) and causes trouble with the dependency resolution?

ScottTodd commented 4 days ago

Have you any other packages installed in your venv that depend on one of the IREE packages (e.g. iree-turbine?) and causes trouble with the dependency resolution?

The workflow runs in https://github.com/nod-ai/shark-ai do have other packages installed, both from a pip cache and from prior steps.

I saw similar, possibly related, behavior on Friday with just iree-base-compiler and iree-base-runtime installed (Discord conversation here). I didn't grab full logs or repro steps though, just this:

Running command to install package:
  pip install --force --upgrade --no-index --force-reinstall --ignore-installed --no-cache-dir --no-deps /home/.iree/bisect/11915893210/iree_base_compiler-3.1.0.dev0+5b0740c97a33edce29e753b14b9ff04789afcc53-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Processing /home/.iree/bisect/11915893210/iree_base_compiler-3.1.0.dev0+5b0740c97a33edce29e753b14b9ff04789afcc53-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-compiler
Successfully installed iree-base-compiler-3.0.0.dev0+4c0fd906bfd1e8888654b3e8a967ef977bb84403

(attempted to install 3.1.0.dev0+5b0740c97a33edce29e753b14b9ff04789afcc53 from a local .whl file, actually installed 3.0.0.dev0+4c0fd906bfd1e8888654b3e8a967ef977bb84403)

ScottTodd commented 4 days ago

Some of the workflows are doing this:

pip install -f https://iree.dev/pip-release-links.html --upgrade --pre \
  iree-base-compiler iree-base-runtime --src deps \
  -e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine"

The source install there is sketchy... that should be more stable once we publish nightly releases of iree-turbine and create a consolidated index page: https://github.com/iree-org/iree/issues/19193

Logs (https://github.com/nod-ai/shark-ai/actions/runs/12054478165/job/33612593692?pr=429#step:5:141):

Looking in links: https://iree.dev/pip-release-links.html
Obtaining iree-turbine from git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine
  Cloning https://github.com/iree-org/iree-turbine.git to ./deps/iree-turbine
  Running command git clone --filter=blob:none --quiet https://github.com/iree-org/iree-turbine.git /groups/aig_sharks/actions-runner-nod-gpu1/_work/shark-ai/shark-ai/deps/iree-turbine
  Resolved https://github.com/iree-org/iree-turbine.git to commit 9e79f4efefb5ba6407dce0f6087eb770e8ad4992
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: iree-base-compiler in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (3.0.0rc20241114)
Collecting iree-base-compiler
  Downloading https://github.com/iree-org/iree/releases/download/iree-3.1.0rc20241127/iree_base_compiler-3.1.0rc20241127-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (71.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.4/71.4 MB 103.7 MB/s eta 0:00:00
Requirement already satisfied: iree-base-runtime in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (3.1.0rc20241127)
Requirement already satisfied: numpy in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from iree-base-compiler) (1.26.4)
Requirement already satisfied: sympy in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from iree-base-compiler) (1.13.1)
Requirement already satisfied: torch>=2.3.0 in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from iree-turbine) (2.3.0+cpu)
Requirement already satisfied: Jinja2>=3.1.3 in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from iree-turbine) (3.1.4)
Requirement already satisfied: MarkupSafe>=2.0 in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from Jinja2>=3.1.3->iree-turbine) (2.1.5)
Requirement already satisfied: filelock in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from torch>=2.3.0->iree-turbine) (3.13.1)
Requirement already satisfied: typing-extensions>=4.8.0 in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from torch>=2.3.0->iree-turbine) (4.12.2)
Requirement already satisfied: networkx in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from torch>=2.3.0->iree-turbine) (3.3)
Requirement already satisfied: fsspec in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from torch>=2.3.0->iree-turbine) (2024.6.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /groups/aig_sharks/actions-runner-nod-gpu1/_work/_tool/Python/3.11.10/x64/lib/python3.11/site-packages (from sympy->iree-base-compiler) (1.3.0)
Building wheels for collected packages: iree-turbine
  Building editable for iree-turbine (pyproject.toml): started
  Building editable for iree-turbine (pyproject.toml): finished with status 'done'
  Created wheel for iree-turbine: filename=iree_turbine-3.1.0-0.editable-py3-none-any.whl size=10268 sha256=275c0c8729333394d8b33bda92863f896bae317bb99669348249de8463f783da
  Stored in directory: /tmp/pip-ephem-wheel-cache-k04ixznh/wheels/c0/80/10/7dfb706f2840d355b7922682cb8daa9a0a3d945044ea22935f
Successfully built iree-turbine
Installing collected packages: iree-base-compiler, iree-turbine
  Attempting uninstall: iree-base-compiler
    Found existing installation: iree-base-compiler 3.0.0rc20241114
    Uninstalling iree-base-compiler-3.0.0rc20241114:
      Successfully uninstalled iree-base-compiler-3.0.0rc20241114
  Attempting uninstall: iree-turbine
    Found existing installation: iree-turbine 3.1.0
    Uninstalling iree-turbine-3.1.0:
      Successfully uninstalled iree-turbine-3.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
shark-ai 3.0.0 requires iree-base-compiler==3.0.*, but you have iree-base-compiler 3.1.0rc20241127 which is incompatible.
shark-ai 3.0.0 requires iree-base-runtime==3.0.*, but you have iree-base-runtime 3.1.0rc20241127 which is incompatible.
shark-ai 3.0.0 requires iree-turbine==3.0.*, but you have iree-turbine 3.1.0 which is incompatible.
shark-ai 3.0.0 requires shortfin==3.0.0, but you have shortfin 3.1.0.dev0 which is incompatible.
Successfully installed iree-base-compiler-3.0.0rc20241114 iree-turbine-3.1.0

So in that case, iree-base-compiler-3.0.0rc20241114 is already installed, as is shark-ai 3.0.0 (which pins to iree-base-compiler==3.0.0). When an install is attempted on iree-base-compiler 3.1.0rc20241127, it fails?

I don't see shark-ai installed or that error message in the other logs though: https://github.com/nod-ai/shark-ai/actions/runs/12021435366/job/33546714200#step:5:224.