airspeed-velocity / asv

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting
https://asv.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
845 stars 177 forks source link

CI: Test on min/max supported ranges of python versions #1366

Closed dstansby closed 5 months ago

dstansby commented 5 months ago

This updates the test matrix to make sure tests are run on every supported Python version. Also updates the Python 3.12 tests to use the released instead of release candidate version.

mattip commented 5 months ago

This adde 3 more runs: python3.8, python3.9, python3.11, and brings the count from 5 run to 8. I think we don't really need that, and it is enough to test all python versions in the wheel builds.

The changes to the pyproject.toml are good, but not really part of this PR (in case anyone thinks of closing this).

dstansby commented 5 months ago

What would be a preferred testing strategy? To test on minimum and maximum Python version (currently 3.7 and 3.12) instead? If time is an issue with the new test runs, note that they run in parallel so shouldn't add any extra testing time.

mattip commented 5 months ago

they run in parallel

While that may be true it is a waste of resources generously donated by github. Is there a particular concern you have with something in asv that may be fragile across python versions?

I would be happy to test with a single version of CPython and a single version of PyPy. When building wheels for a release, we build the package for all version of python, so in the unlikely case of version-specificy breakage, the problem will be caught then.

dstansby commented 5 months ago

Certainly for new versions of Python there are removals and deprecations, so it would be nice to test that asv is working on the latest version of cPython. And then my thinking is it's good to test against the oldest version too, to prevent someone accidentally using API introduced in e.g. Python 3.11 that would break support for Python 3.7.

HaoZeke commented 5 months ago

Thanks @dstansby 🚀