Build and publish Python 3.13 and 3.13t wheels

ScottTodd commented 2 weeks ago

Request description

Python 3.13 is currently in prerelease and will be released soon. It also includes a "free threaded" (no Global Interpreter Lock) variant. We should start building the iree-compiler and iree-runtime packages for each of these versions.

What component(s) does this issue relate to?

Python

Additional context

For Python 3.12, see https://github.com/iree-org/iree/issues/15856.

We are starting to use Python 3.13 and free-threaded support downstream in the shortfin package/project under https://github.com/nod-ai/SHARK-Platform/ (see https://github.com/nod-ai/SHARK-Platform/issues/130).

For free-threaded support, we'll likely need to update and/or change our usage of pybind11 (for the compiler) and nanobind (for the runtime)

ScottTodd commented 2 weeks ago

We could wait until Python 3.13 is fully released (on 2024-10-07), but https://www.python.org/downloads/release/python-3130rc3/ actually encourages early adoption:

We strongly encourage maintainers of Python projects to prepare their projects for 3.13 compatibilities during this phase, and where necessary publish Python 3.13 wheels on PyPI to be ready for the final release of 3.13.0. Any binary wheels built against Python 3.13.0rc1 and later will work with future versions of Python 3.13.

ScottTodd commented 2 weeks ago

I was able to build release packages on Linux (manylinux_2_28) for Python 3.13 and 3.13t with some dockerfile updates: https://github.com/iree-org/base-docker-images/pull/15. No changes to IREE source code or our pybind/nanobind deps needed. Not sure if the build on its own is enough though, may need to run some tests too.

ScottTodd commented 2 weeks ago

I have a branch nearly ready to test: https://github.com/ScottTodd/iree/tree/infra-python3.13

Need to land https://github.com/iree-org/base-docker-images/pull/15 first to get Python 3.13 in our manylinux dockerfile.

Building for more versions will increase nightly release CI time, and we should be testing against these versions too (see https://github.com/iree-org/iree/issues/15856#issuecomment-1947134679 and other comments there).

ScottTodd commented 2 weeks ago

Got a mostly successful test release build last night: https://github.com/iree-org/iree/actions/runs/11156205488 using this branch: https://github.com/iree-org/iree/tree/users/scotttodd/python3.13.

I tried installing python 3.13t on macOS and Windows but need to do more troubleshooting. Each installer requires some extra options for the free-threaded support and puts the binary on a different path, so the existing scripts need some changes:

ScottTodd commented 2 weeks ago

Stashing some work before vacation / if someone else wants to help or follow along:

Branch with script changes / debugging: https://github.com/ScottTodd/iree/tree/infra-python3.13
- For 3.13t, see https://docs.python.org/3.13/using/mac.html#installing-free-threaded-binaries and https://docs.python.org/3.13/using/windows.html#installing-free-threaded-binaries. There are scripts floating around in GitHub issues linked on that branch to set the Include_freethreaded option on Windows and -applyChoiceChangesXML ./choicechanges.plist on macOS
- Debugging via github actions is slow. Next step is to run the scripts locally to debug what they are doing and where the packages are going on disk
Testing the runtime release builds with compiler builds and arm64 builds commented out on my fork using workflow_dispatch of https://github.com/ScottTodd/iree/actions/workflows/oneshot_candidate_release.yml, then watching the builds at https://github.com/ScottTodd/iree/actions/workflows/build_package.yml and checking the uploaded release artifacts at https://github.com/ScottTodd/iree/releases
- This process should work for anyone, on their fork.

marbre commented 2 days ago

The (minimal!) changes that are need for IREE itself are in #18770.

iree-org / iree