search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.51k stars 3.02k forks source link

Pip releases should test against all versions of Python for Windows #13011

Open notatallshaw opened 1 week ago

notatallshaw commented 1 week ago

What's the problem this feature will solve?

Currently for regular PRs pip only tests against the oldest supported Python version and the newest supported Python version: https://github.com/pypa/pip/blob/ec5faeac4ef6bae97df0e779566ceb2b0de89d3f/.github/workflows/ci.yml#L175-L182

But when pip does a final release I think it would be reassuring for all versions of Python to be tested against.

Describe the solution you'd like

Somehow in the GitHub workflow file have a condition that runs all untested versions of Python for the final release.

Alternative Solutions

Do nothing.

Additional context

I don't know how to implement this yet, but if no one else does this I plan to spend a bit of time investigating this in the future and raise a PR. I'm guessing have a job associated with a specific path filter on NEWS.rst that tests these versions of Python on Windows.

Code of Conduct

sbidoul commented 1 week ago

The release process creates a branch named release/YY.N(.P), to using that to determine that more tests should run could be an option. Although as a RM I wouldn't want to discover issues at that stage.

Maybe a scheduled run of the pipeline with more tests enabled instead?

pfmoore commented 1 week ago

Although as a RM I wouldn't want to discover issues at that stage.

Agreed. I think if we want to test against all Python versions, we should be doing that sufficiently before the release that we have the time to assess and fix any issues before starting the release. And in particular, anything that makes it the RM's responsibility to deal with failures here isn't reasonable - we recently changed policy to merge vendoring updates well in advance of a release in order to remove the risk that they cause issues with the release. Any testing of all Python versions should follow the same principle.

Maybe a scheduled run of the pipeline with more tests enabled instead?

That sounds like a better idea. Maybe run it quarterly, in the month before each release is due?

notatallshaw commented 1 week ago

There's already a weekly scheduled test run:

https://github.com/pypa/pip/blob/ec5faeac4ef6bae97df0e779566ceb2b0de89d3f/.github/workflows/ci.yml#L11-L12

Assuming there's a way to distguish that in the workflow file, seems like this would be a reasonable place to add these test runs based on feedback.

pradyunsg commented 1 week ago

We'd dropped this because Windows CI was exceedingly slow and we were hitting issues due to low quotas for the CI pipelines. I think we can reasonably remove that limitation now.

ichard26 commented 1 week ago

IIRC we're on a [much larger] organisation plan now. Perhaps it would be easiest to run the full suite of Windows jobs across the entire Python version matrix. I'm a little worried that weekly runs may have poor visibility (AFAIK it's only maintainers that get notifications if they fail).