search

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.
http://iree.dev/
Apache License 2.0
2.86k stars 622 forks source link

PyPI package promotion script is not pruning rc suffix from compiler wheels #19155

Closed ScottTodd closed 1 week ago

ScottTodd commented 1 week ago

What happened?

The https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/promote_whl_from_rc_to_final.py script used by https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/pypi_deploy.sh is changing some of the package version metadata, but not enough for pip. The 2.9.0 release of https://pypi.org/project/iree-base-compiler/ is reported as version iree-base-compiler==2.9.0rc20241108

Steps to reproduce your issue

See the issue using stable pypi releases:

python3.11 -m venv 3.11.venv && source 3.11.venv/bin/activate
python -m pip install iree-base-compiler

pip freeze
# iree-base-compiler==2.9.0rc20241108
# mpmath==1.3.0
# numpy==2.1.3
# sympy==1.13.3

Reproduce locally:

  1. Download a wheel file from https://github.com/iree-org/iree/releases/tag/iree-3.0.0rc20241114

  2. Run the promotion script on a single release:

    cd build_tools/python_deploy
python3.11 -m venv 3.11.venv && source 3.11.venv/bin/activate
python -m pip install -r pypi_deploy_requirements.txt
./promote_whl_from_rc_to_final.py ~/Downloads/iree_base_compiler-3.0.0rc20241114-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

  3. Check the version on the converted file:

    python3.11 -m venv 3.11.venv && source 3.11.venv/bin/activate
# Note the lack of 'rc20241114' here, this is the "promoted" whl file
pip install ~/Downloads/iree_base_compiler-3.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
pip freeze

# iree-base-compiler==3.0.0rc20241114
# mpmath==1.3.0
# numpy==2.1.3
# sympy==1.13.3

What component(s) does this issue relate to?

Python

Version information

Tip of tree: https://github.com/iree-org/iree/commit/d4975713a0aa3ba872807fc16585fb4b5a04e41d

Additional context

Looking into the .whl file (they are zip archives), I see:

The runtime wheel and our downstream wheels like https://pypi.org/project/shortfin/ don't have that .egg-info folder or PKG-INFO file

So next step is figuring out what .egg-info is, why we are producing it, and if https://github.com/hauntsaninja/change_wheel_version or our own scripting should handle it. I can also directly edit the package files to see if that is the cause.

ScottTodd commented 1 week ago

We also embed a version.py file in the package using this code: https://github.com/iree-org/iree/blob/d4975713a0aa3ba872807fc16585fb4b5a04e41d/compiler/setup.py#L307-L318

Within the .whl file, that ends up looking like /iree/compiler/version.py with contents:

# Auto-generated version info.
PACKAGE_SUFFIX = ""
VERSION = "3.0.0rc20241114"
REVISIONS = {"IREE": "eef2c3a1544a7e0d772dc9a480ecbcb00acd0604"}

That may or may not be related to what pip reports for the package version. I'll edit the archive to test.

ScottTodd commented 1 week ago

Editing the version embedded in iree_base_compiler.egg_info/PKG-INFO changes the version reported by pip freeze.

I found the code in change_wheel_version that edits .dist-info/METADATA: https://github.com/hauntsaninja/change_wheel_version/blob/e28436ebdb6fe5a81cd29922f7276d00675d1bd3/change_wheel_version.py#L128-L144 . Could add similar code for .egg-info/PKG-INFO, or see if we can remove that .egg-info folder in the wheel. I see things online suggesting that egg files/directories may be deprecated? https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html Is there some version upgrade or packaging change that we could make that would remove those files entirely?

Image

ScottTodd commented 1 week ago

This may be related to the setuptools version, so I checked the logs:

Inserting a pip install --upgrade pip here does not help: https://github.com/iree-org/iree/blob/d4975713a0aa3ba872807fc16585fb4b5a04e41d/build_tools/python_deploy/build_linux_packages.sh#L191 I saw posts online suggesting updating both setuptools and pip itself for issues relating to egg_info, e.g. https://askubuntu.com/questions/975523/pip-install-gives-command-python-setup-py-egg-info-failed-with-error-code-1

ScottTodd commented 1 week ago

I'm still trying to figure out why the iree_base_compiler.egg-info folder exists at all, when iree_base_compiler-3.0.0.dist-info already exists. Maybe we can just delete the .egg-info from the .whl entirely...? It would be better to never produce it in the first place, or fix the metadata in there.

ScottTodd commented 1 week ago

Still not sure why these files are being produced, but I'll try scripting a way to edit the version in .egg-info/PKG-INFO file I guess 🤷

ScottTodd commented 1 week ago

Found some clues on https://stackoverflow.com/questions/3779915/why-does-python-setup-py-sdist-create-unwanted-project-egg-info-in-project-r

This directory is created intentionally as part of the build process for a source distribution.

You may safely delete the directory after your build has completed.

An alternative solution to jathanism's method could be the usage of the egg_info hook. I use it to clean up before each build:

ScottTodd commented 1 week ago

Another clue: the CMake build that setup.py kicks off creates a few iree_base_compiler.egg-info folders, which then appear to get copied in to the .whl file:

Image

Our other packages don't use CMake "install" from what I can tell, they just "build"?

ScottTodd commented 1 week ago

Well that was a "fun" three hours of learning more about Python packaging. Fix PR is landing: https://github.com/iree-org/iree/pull/19156.