Open hugovk opened 1 year ago
Hmm, and Azure Pipelines is only running for PRs to main
(not backports); and on the version branches after PRs have been merged, but not for 3.12
.
There are two top-level Azure Pipelines config files:
This runs on version branches, after a PR has been merged.
But when we forked 3.12 from main
in May, we forgot to add '3.12'
to the config, so it's not run on that branch at all:
trigger: ['main', '3.11', '3.10', '3.9', '3.8', '3.7']
This runs when PRs are opened. But it's only running for PRs against main
, not for any of the backport PRs. I'm not sure why.
To illustrate, this set of PRs only only had AP builds for main
(python/cpython#109360) and not 3.12
(
python/cpython#109361) or 3.11
(https://github.com/python/cpython/pull/109362).
After merge, AP ran for main
(https://dev.azure.com/Python/cpython/_build/results?buildId=137654&view=results) and 3.11
(https://dev.azure.com/Python/cpython/_build/results?buildId=137657&view=results) but not 3.12
.
For comparison, GitHub Actions runs for PRs for all branches, but not merged pushes to version branches.
I believe @zooba said there were some things being tested on Azure Pipelines that GitHub Actions didn't support?
Azure Pipelines is doing install layout testing using the APPX layout as a second step post-commit. While I'd most like to have the default build match the install layout, this has been a useful way to catch issues in tests that assume a source directory layout. These tests are templated at https://github.com/python/cpython/blob/main/.azure-pipelines/windows-layout-steps.yml and included a few times in ci.yml
.
Looks like we could potentially remove the "Tests" step (and "Publish Test Results") from windows-steps.yml
without affecting the layout testing? The main thing in windows-steps.yml
being the CPython build.
In addition to the APPX layout step, should we also keep the nuget and embed layout steps?
Yeah, that would work. But it's post-commit testing anyway, so it isn't holding anyone up. And knowing that the regular tests pass is incredibly valuable when something specific to one of the layouts fails.
I'm pretty sure people are only annoyed by the PR integration. If that goes away, nobody will care unless they're looking for it.
As for the different layouts, the nuget and embed layouts only do really quick tests. I'd keep them all, as it gives us early warning that something has broken them (e.g. a new DLL isn't being copied). Otherwise we won't find out until during the release process when the same tests are run.
Sounds good. I suggest we proceed as follows:
ci.yml
be, only change pr.yml
pr.yml
, keeping patchcheck
on APpatchcheck
to GHA, either as is or equivalent lint checkspr.yml
Clarification on 1:
ci.yml
be, change Windows testing in pr.yml
Let's remove Ubuntu unit tests from both ci.yml
and pr.yml
, because they're run on GHA.
That leaves only patchcheck
in pr.yml
(it's skipped in ci.yml
because it only makes sense for PRs).
Please see PR https://github.com/python/cpython/pull/109452.
GitHub Actions is easier to use and maintain. For example, GitHub Actions is more popular and widely used, and so more familiar. It's easier to maintain a single system. Also core devs can restart failed GitHub Actions whereas they can't for Azure Pipelines. Both are owned by Microsoft and I believe they run on much the same infra.
There's a lot of duplication of tests between Azure Pipelines and GitHub Actions.
Let's remove the duplicated tests, and move the unique things over to GitHub Actions. For example,
patchcheck
is only run on AP.We might not necessarily need to port things directly over, it may be possible to replicate equivalent checks in another way, possible with a pre-commit or Ruff lint rule. Let's evaluate these separately.
Related issues/PRs:
Linked PRs