Build Quality Checks issue: incorrect baseline detection

[sorbonad]

Hello, I would like to report a problem: we want to execute quality checks to verify that current code coverage is improving from previous build for a specific branch (or same branch), but after testing multiple configuration we are seeing that this is always checking against the last good build which could belong to the same branch or not, so if multiple branches with different code coverage levels are producing false negatives in our pipeline. The code is hosted in bitbucket on promise, we have azure cloud and 2 building agents on promise. Could you help us please?

[ReneSchumacher]

Hi @sorbonad,

sorry for the inconvenience caused by this issue. It seems to be related with the external code repository as we have seen a similar report where the code was hosted on GitHub. I'll start analyzing the behavior when I'm in the office tomorrow and hopefully have a fix in the next release which we're already preparing.

As soon as I have more information I will post an update here with the root cause and he timeline for the fix.

René

[ReneSchumacher]

I already found the root cause and fixed it. Unfortunately, we forgot to add proper support for external code repositories. Luckily, this was a quick fix. The fixed version of the task should be coming either tomorrow or on Tuesday. Stay tuned.

René

[ReneSchumacher]

Hey @sorbonad,

a new task version has been published that fixes this issue. Please let me know if you still have problems with external source repos.

Happy building, René

[sorbonad]

@ReneSchumacher I will test and will keep you posted

[sorbonad]

Hi @ReneSchumacher, I am testing and I am founding that if you set the Baseline section pointing to a specific Build definition, Repository and Branch, the found build ("Found baseline build with ID xxxx") is the first build of the story of that branch one instead of the last one. If you leave those properties empty seems to be working fine. I am still verifying. regards,

[ReneSchumacher]

@sorbonad: You don't need to specify definition, repository, and branch unless you want to compare against a build that ran for a specific branch other than the current branch, or you want to compare against a totally different build (i.e., different build definition).

When task runs is fetches the definition and branch of the build that is executing the task. It then looks for the latest (ordered by finish time descending) build that ran for the same definition and branch before the current build was started. In other words: if you run a build for definition 1 on branch master at 1 pm, it searches for a build from definition 1 that ran on branch master and finished before 1 pm. This ensures that policies always compare values from the same build for the same branch.

If you specify definition, repository, and branch, the task queries for builds of that particular definition and branch that finished before the current build's starting time. You usually use this for pull request builds, where most people want to compare against values from the latest build on master.

We also only compare to successful builds unless you configure the policy to include partially succeeded builds as well.

[sorbonad]

Hi @ReneSchumacher, I can confirm that if you leave the Baseline properties empty it is working as expected (this work for us and is good enough to continue working). If you set a specific build definition/repo/branch it is taking the first build, probably it works for the scenarios you explained. Thanks for the prompt release of this fix!

[ReneSchumacher]

I cannot reproduce that behavior you describe. Even if I configure the baseline parameters, it still finds the correct baseline build (always the latest). Since the task calls the same API in both cases, the order of the search cannot change.

If you want you can send me the log files of the BQC task from the correct and the incorrect build to PSGerExtSupport@microsoft.com and I'll have a look.