Automate build and release to PyPI

[hugovk]

Currently the release process has many steps, including preparing things in this repo, and then starting a build in https://github.com/python-pillow/pillow-wheels, before downloading the sdist + wheel artifacts to disk, and manually uploading them to PyPI using twine.

I'd like us to move towards more automated releases, including using the new Trusted Publishers to upload wheels directly from the CI to PyPI, which is more secure:

• https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/

We can do this in a phased manner, not all of these things would need to be done at once, to ensure we can still make releases with just some of them completed.

We could potentially deal with the Travis CI wheels later, and tackle GitHub Actions first as they're a bigger win and we don't know when Travis CI will go away.

Big picture:

• Prepare release
• Push a tag
• The CI builds release artifacts
• The CI uploads artifacts to PyPI

I'd suggest something along these lines:

1. Move Linux and macOS https://github.com/python-pillow/pillow-wheels to this repo https://github.com/python-pillow/Pillow. We already have Windows wheels build here.

   • We'll need to add paths: / paths-ignore to workflows so we only trigger the main test workflows for "normal" code changes, and only trigger the wheel workflow for things like release and modifying its config
   • Upload wheels as GitHub artifacts in this repo instead

2. Add automation to download artifacts and upload them to PyPI via Trusted Publishing

   • This may need refactoring to move all the wheel building to the same workflow (but separate jobs), and then a final job that runs when the wheels have all been built, that builds sdist, checks and uploads them all.
   • See https://github.com/ultrajson/ultrajson/blob/main/.github/workflows/deploy.yml and https://github.com/ultrajson/ultrajson/actions/workflows/deploy.yml for examples.
   • We also have https://test.pypi.org/project/Pillow/ which we can use to verify the upload machinery is working smoothly before release. For example, we can upload for non-tagged wheel builds.

Thoughts?

[wiredfool]

This sounds like a great step.

[aclark4life]

Love it!!

[radarhere]

I've created https://github.com/python-pillow/Pillow/pull/7418 for 1.

[hugovk]

I've opened https://github.com/python-pillow/Pillow/pull/7544 to create the sdist during the release build.

---

Next, I'd like us to move the Windows wheel building from test-windows.yml into the same wheels.yml workflow.

So we get something like this:

And the Windows wheels should be part of the same dist.zip artifact containing the Linux and macOS wheels, and the sdist.

I think we also probably want to use a reusable workflow, similar to wheels-linux.yml and wheels-macos.yml.

We don't need to build the wheels anymore in test-windows.yml. If there's a lot of common steps needed for testing (in test-windows.yml) and building wheels, a reusable workflow could be useful here.

This doesn't all need to be done in one go, incremental steps are fine, as long as they're atomic changes and we're still able to make releases during any intermediate step.

Any thoughts on how to approach this? cc @nulano

---

After all this is done, we have all* the release files created in one place on the CI, and do the upload to PyPI! But one step at a time :)

(* All the GHA files, not Travis CI, which will still need manual upload, but they should be ready before all the automation.)

[nulano]

I've managed to use cibuildwheel to build wheels locally, I should be able to make a PR on top of #7552 for Windows wheels once it is merged. Edit: I've got a WIP branch ready, including ARM64 wheels (although I have no way to test them!). Edit2: I've created PR #7580.

[hugovk]

Sounds good! When the time comes, we can ask around for someone to test the ARM64 wheels :)

[nulano]

I've since realized that I can borrow a relative's M2 MacBook to test the wheels. The Python 3.12 wheel from my WIP branch tested OK. I then used cibuildwheel manually in the terminal to build and test wheels for the 10.1.0 tag for https://github.com/python-pillow/Pillow/discussions/7547#discussioncomment-7592240

[radarhere]

7616 has now been merged.

@aclark4life could you go to https://pypi.org/manage/project/pillow/settings/publishing/ and add a publisher?

Owner: python-pillow Repository name: Pillow Workflow name: wheels.yml Environment name: release-pypi

[hugovk]

@aclark4life Please can you also set up an environment named release-pypi in this repo, that all the release managers can access: at least @radarhere and me, probably also you and @wiredfool for backup.

[aclark4life]

@radarhere @hugovk Thanks and done, I think! The env release-pypi already exists … need me to do anything to it?

[hugovk]

Thanks, the publisher looks good!

The env release-pypi already exists … need me to do anything to it?

@aclark4life What settings does it have?

@radarhere What settings should it have?

[aclark4life]

Most notably, no environment vars or secrets configured.

[radarhere]

The publisher looks good to me, thanks. The details I mentioned were the only ones I added.

[hugovk]

So when releasing, after pushing the tag, the CI will begin, build the sdist and all the wheels, and somehow put them in an environment? How does that look? What's the next step to upload to PyPI?

[aclark4life]

Probably need a testpypi release first.

[radarhere]

When I was using Test PyPI in #7616, once sdist and the wheels were built, the action uploaded them, and that was it.

https://github.com/radarhere/Pillow/actions/runs/7329629479/job/19960359962#step:4:1008

View at: https://test.pypi.org/project/pillow/10.2.0.dev1/

I presume that Test PyPI works the same way as real PyPI. I would have imagined that real PyPI releases don't rely on test PyPI to go first, since we've been ignoring test PyPI for every previous release we've done.

[hugovk]

Okay, so the environment isn't doing anything. Anyway, I'm excited for the next release!

@radarhere Please will you update https://github.com/python-pillow/Pillow/blob/main/RELEASING.md? It can wait until after doing the release if you like.

[radarhere]

Sure. I've created #7652

[radarhere]

The tagged "Upload release to PyPI" ran smoothly in the Pillow 10.2.0 release.

With that, is this completed?

[hugovk]

Great to hear it went smoothly!

Before closing this issue, we still have Travis to automate. Shall we try my suggestion at https://github.com/python-pillow/Pillow/issues/7485#issuecomment-1873821482 to upload to PyPI directly from Travis?

[radarhere]

Sure, sounds good if it works.

[hugovk]

Plan C: please see PR https://github.com/python-pillow/Pillow/pull/7690 to replace Travis native builds with GHA emulated builds.

[aclark4life]

Only loosely following, but I think I like Plan C!! Thanks all ❤️ (Losing travis in exchange for longer build times is my "hot take", sounds worth it to me.)