`maturin generate-ci github --pytest`: non-x86 architectures stuck on Python 3.10 for pytest

matt-codecov commented 1 month ago

Bug Description

The problem

I am trying to publish a package with requires-python = ">=3.12" set in pyproject.toml and am having trouble with maturin's generated CI workflow (specifically trying to run pytest).

When the target matrix value is x86 or x86_64, pytest is run with the runner's Python version which, thanks to actions/setup-python, is 3.12. x86 and x86_64 thus work fine for my 3.12-only package.

However, when the target matrix value is anything else, the job uses uraimo/run-on-arch-action to create an Ubuntu 22.04 image of the appropriate architecture to run pytest on. That image installs the latest version of Python in the stock Ubuntu 22.04 apt repository which is 3.10. My package requires 3.12, so the maturin action only built wheels for 3.12, so the job fails with an error like:

Looking in links: dist
  ERROR: Could not find a version that satisfies the requirement codecov_rs (from versions: none)
  ERROR: No matching distribution found for codecov_rs

(link to an actual run)

Workaround

https://github.com/codecov/codecov-rs/pull/55 makes the uraimo/run-on-arch-action step set up the deadsnakes PPA, install Python 3.12 from it, and then set up a virtual env so pip will work. Here's a passing run from a test PR that took this approach.

If that sounds like a good generic solution I can take a crack at opening a PR. Looks like it'd just be some changes in src/ci.rs? It doesn't look like there is a "latest" package in the deadsnakes PPA though so the hardcoded 3.12 version number would need to be updated down the line.

Your maturin version (`maturin --version`)

1.7.1

Your Python version (`python -V`)

3.12

Your pip version (`pip -V`)

pip 24.0

What bindings you're using

pyo3

Does `cargo build` work?

[X] Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash `/`)?

[X] Yes

Steps to Reproduce

Write a Python package with requires-python = ">=3.12" set in pyproject.toml.
Run maturin generate-ci github --pytest > .github/workflows/publish.yml
Modify .github/workflows/publish.yml to remove Windows/macOS/musllinux, remove the publishing step, and trigger on PR (or however you'd like to test)
Trigger the workflow
Observe that the Linux pytest step passes for x86 and x86_64 (good)
Observe that the Linux pytest step fails for aarch64 and probably other architectures (bad)

messense commented 1 month ago

Looks like it'd just be some changes in src/ci.rs?

Yeah, pull requests are welcome!

mhammerly commented 1 month ago

(same person, personal account)

I just noticed GitHub has started migrating ubuntu-latest to 24.04 which has Python 3.12 in its stock repository so I put a trivial PR up for that.

Python 3.13 will be out soon and I don't know whether 24.04 will update so I'll still see about getting a PR up for the deadsnakes approach later. I think https://github.com/codecov/codecov-rs/pull/55 could be improved before merging:

Moving the Ubuntu version, Python version, and PPA stuff into consts with doc comments
Seeing if it's possible to detect when the deadsnakes stuff is actually necessary. Installing the prerequisites slows things down a bit and if the Python version we ultimately install is the one from the stock repository it's wasted work

I'll try to follow up, but if any other community members want to jump in that'd be welcome :)

messense commented 1 month ago

@mhammerly Had to revert #2231 because there is no 24.04 image support in run-on-arch-action yet.

mhammerly commented 1 month ago

sorry about that! I just noticed that myself and was going to close the PR.

jonaspleyer commented 4 weeks ago

I have encountered the same error with only python>=3.8. Thanks for already working on this problem. I would love to use pytest in the future again. In case you are interested in the error messages: https://github.com/jonaspleyer/cr_mech_coli/actions/runs/11411487573

PyO3 / maturin