[BQC] multiple checks with status policies

[mjvankampen]

From the documentation I am not 100% clear on what happens if we have multiple checks. For example when I run a PR the following is checked:

• Code coverage
• Documentation coverage (similar to line coverage)
• Compiler warnings
• PC-lint warnings
• CppCheck warnings

Does the coverage policy status and warning status policy fail when any of these checks fail? Or does it pick the first one? I guess the sentence below is not completely clear to me.

Unlike with the simple build validation described above, you can run multiple Build Quality Checks policies (e.g., enabled warnings and code coverage policies) but only check one of them in your pull request (e.g., only block pull requests based on code coverage).

Ps. policy status is a great addition, improves CI a lot!

[ReneSchumacher]

Hi again,

I guess I should send out the docs to multiple colleagues to make sure it's clear. The important part is described in the policy documentation (e.g. https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/CodeCoveragePolicy.md#pull-request-status).

When you run multiple BQC tasks without explicitly setting their run title to something unique, all task instances publish the status with the same name (e.g., bqc/warnings-policy). Since we naturally run into this scenario for multi-config builds, PR statuses are only updated if the current status is either pending or succeeded or if the status "severity" increases. I.e., a failed status will always overwrite a pending or succeeded status, and an error status will always overwrite a pending, succeeded, or failed status. Thus, if any one of the policies fails, the PR status should be failed.

If you want to distinguish between the different checks, please set the run title and configure the pr status policy with the name bqc-{runTitle}/{policyName} and replace any whitespace in the runtitle with dashes.

[mjvankampen]

Ah very clear missed that in the documentation, I was looking at the pull request tutorial. That piece of documentation is completely clear!