Pull Request Policy Builds undocumented

[binaryunary]

I'm interested in applying the Build Quality Checks task to our PR build policy, but I can't find any info on how to make it work for this use case. This piece of documentation is still in a TODO state: https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#pull-request-policy-builds.

What I'd like to have is to base the quality policy on previous builds of a specific build definition, regardless of the branch, because with PRs the temporary branches are dynamic and change all the time. Is this use case supported?

[ReneSchumacher]

Hi!

My apologies for the missing documentation, I still haven't gotten around to create something. The scenario is fully supported and we're actually using it in our own pipeline:

The Build Quality Checks needs to be configured with the correct baseline build (see screenshot). Simply select the build definition, repository and branch to use (in our case we always compare to the latest build on master).

I usually recommend having a single build definition that is used on master as well as all PRs. If there are steps that should not run during a PR build, you can use task conditions to skip specific build tasks as shown in the screenshot below.

I hope this short description helps. If you need anything else, just let me know.

Happy building, René

[binaryunary]

Thank you for the quick reply.

This is actually what we have been doing so far - a single build definition with task conditions.

The reason we're considering dedicated 'verification' (build, test, check quality) build definitions is that we've moved to a monorepo and we have organized our directory structure by products (solutions). Each product contains multiple deployable components/services. Instead of having to add build policies with correct trigger filters for each component it is simply easier to have a single verification build definition per product with broad filters.

So I was wondering whether it is somehow possible to set up the quality check task to keep track of all builds of a particular build definition.

In the documentation it says:

Select the build definition that should be used to search for the baseline build. If you do not set a value, the last build of the current build definition will be used when comparing policy values.

Which seems to be what I'm looking for - each new build would check previous build to make sure there are no quality regressions (regardless of the targeted branch), but so far I have been just getting: The baseline build could not be found. All policies based on the previous build will pass..

If it is not possible then I guess one workaround could be scheduling nightly builds against master and then use this as baseline for PRs.

[ReneSchumacher]

Let me make sure that I correctly understood your setup:

• You have a single repo with multiple products/components that need to be built independently.
• You have one build definition per product/component that probably gets triggered based on path filters.
• You are using topic branches and pull requests to integrate new stuff into each product/component.
• You want to compare code coverage on each PR with the previous value to make sure there is no decrease.

When you configure the Build Quality Checks task to compare against a previous build, it automatically looks for builds from the same build definition that ran for the same branch as the current build. For PRs, this is a problem because PRs use temporary branches created by the build that always look like this: refs/pull/<PR ID>/merge. Thus, the first build in a PR will never find a baseline build as no build ever ran for this temporary branch. Subsequent builds then compare to the previous build in the same PR, which is usually not what you want because the initial build might have had a much lower code coverage than a build of the unchanged code.

Thus, you need a common baseline for all PR builds and that usually is a build based on the master branch. In most cases, I tell customers to create a build definition with a CI trigger (and in your case an appropriate path filter) and use that same build in the branch policy. If you have multiple builds for the same repo, use the same path filters in your branch policy like this:

With this setup, every integration of a topic branch into master (i.e., the completion of a PR) will trigger your build on master and set the new baseline for all other PRs.

Does that make sense?

[binaryunary]

Yes, your understanding is correct. And what you suggested seems like the more correct solution to my problem. I will probably switch to this approach once our code structure is more stable, but I'll just use these verification build defs for now and will use nightly builds against master as baseline.

Thank you for taking the time to help me.

[ReneSchumacher]

Hey,

good to hear that you solved the problem with runsettings. I'm closing the issue now, but if you need anything else, please feel free to reopen it, open another one, or send an email to PSGerExtSupport (at) microsoft.com.

Happy building! René