BuildQualityChecks doesn't use the expected baseline

[nickduch]

Describe the context

• Extension: BuildQualityChecks
• Environment: Azure DevOps Services (cloud)
• Agent type: Microsoft-hosted
• Pipeline type: yaml pipeline

Describe the problem and expected behavior I'm trying to add BQC to my PR builds and enable the CheckWarnings and CheckCoverage options. When looking at the logs and what happens, it looks like the extension isn't selecting the appropriate master build to use as a baseline. I got two example from two different pipelines: 1. 2.

Here's the configuration for my BCQ task:

Everything else looks fine. By that I mean that the tasks does get the code coverage from both the PR build and the baselines and the check outcome is as expected. However I'm having a hard time understanding the logic here, why are my most recent and sucessful builds getting ignored as the baseline?

[ReneSchumacher]

Hi @nickduch,

sorry for the delay, I have been on vacation until yesterday. Let me first rephrase what you're trying to achieve to see if I really understood the issue here: You have configured a PR validation build in your branch policy, which contains BQC with the configuration you showed above. I.e., you want all PR build results compared with the latest run (of the same definition!) on the master branch, regardless of which branch is targeted by the PR. Is that correct?

At least, this is how the task behaves based on your configuration. Baseline build are selected based on the following criteria:

• The build must be from the same build definition as the PR validation build (otherwise, you must set the parameter baseDefinitionId).
• The build must be completed (this is why the two running builds in your first picture are not considered).
• The build result must be either succeeded or partially succeeded (this is why the two failed builds in your first picture are not considered).

From the screenshots, I cannot understand why the second last build in the first picture and the second build in the second picture are not considered as baselines. But before I dig deeper, please confirm that you only have on build definition that is being used for both the individual or CI builds on master and your PR validation build.

Judging from your pictures, you seem to have a more complex CI/CD pipeline including approvals. We have never really put much thought into this scenario. However, in such a case, looking for completed pipelines most likely does not yield the correct outcome. I'll put work on our backlog to consider running pipelines in which the job containing BQC is already completed. Otherwise, pipelines that have a successfully completed build stage but are still waiting for a deployment approval are not considered as baselines. Thanks for (indirectly) pointing that out! 😄

[nickduch]

Hey @ReneSchumacher No worries and I hope you had a nice vacation :) We do use the same build definition for the "Individual CI" and "PR" builds. The different stages you see in my screenshot are the results of conditions on the different stages. For example, the "production stages" are being skipped if we are not building the main branch.

I had a feeling that those build "waiting for approval" were being ignored, even though I couldn't find any indication regarding this scenario in the documentation, This is why I chose another pipeline for my second example (the second screenshot) with all the stages completed (or skipped). I'm glad you are considering the changes to include pipelines waiting for approval 👍.

Let me know if you need me to provide more info.

[nickduch]

I've just upgraded to v9.0.0, I saw it was released a couple of weeks ago. Initially, I was using v8.2.1.

Sadly, I'm reporting the same issues.

It skipped what seemed to be a good build.

This is from the debug logs from my PR build where we can see its gonna use build 876244 as baseline.

[ReneSchumacher]

Hey,

the baseline behavior hasn't changed in version 9.x, so I'm not surprised that you see the same behavior. I just double checked the code and realized that I forgot one more rule for selecting the baseline build. We only look for builds that finished before the current build started. This usually made sense, but we must let go of that as well if we want to consider finished jobs instead of finished builds.

Looking at the dates and runtimes in your screenshot, that doesn't explain why build #20230526.1 was not considered, though. Can you try running the following query in the browser and send me the result (maybe via email to PSGerExtSupport@microsoft.com):

https://dev.azure.com/stratocast/ClearID/_apis/build/builds?definitions={definitions}&statusFilter=completed&resultFilter=succeeded|partiallySucceeded&maxBuildsPerDefinition=10&deletedFilter=excludeDeleted&queryOrder=finishTimeDescending&api-version=7.1-preview.7

Just replace the definitions parameter with the ID of your build definition.

[nickduch]

The email with the logs have been sent

[nickduch]

So based on that query I ran, I can guess that BQC takes the first master build based on the latest finish time? the build that got skipped in my screenshot also had an approval tasks which timed out after 30 days. This delay added extra days to the finish time which isnt visible on the azure devops UI because they probably list the pipelines ordered by start time instead. So I think I got this figured out. Can you validate my assumptions?

Also, if you do work on including builds with pending approvals, I would be very much interested and that would have solved much of these questions I had. :)