UpperThreshold not taken into account if increase in coverage

[jrperron88-zz]

What i'm trying to accomplish: Error if total coverage is below 90% or code coverage has decreased (even if over 90%). I had accomplished this with two seperate checks - one for fixed total coverage percent, and one for increase in coverage. From the documentation it appears that I should be able to add coverageUpperThreshold to achieve this. But it seems to only care about the coverage increasing

  - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
    displayName: 'Check build quality'
    inputs:
      runTitle: 'IncrementalCoverage'
      checkCoverage: true
      forceCoverageImprovement: true
      coverageUpperThreshold: 90
      coverageType: branches
      ignoreDecreaseAboveUpperThreshold: false
      baseDefinitionId: XXX
      baseRepoId: XXX
      baseBranchRef: refs/heads/master
    continueOnError: true

here is the full log of that execution

  Evaluating coverage data from 1 filtered code coverage data sets...
  Total branches: 18
  Covered branches: 16
  Code Coverage (%): 88.8889
  Found baseline build with ID 328729.
  Successfully read code coverage data from build.
  Evaluating coverage data from 1 filtered code coverage data sets...
  Total branches: 20
  Covered branches: 16
  Code Coverage (%): 80
  [SUCCESS] Code coverage policy passed with 88.8889% (16/18 branches).

[ReneSchumacher]

Hi @jrperron88,

unfortunately this cannot be done with a single instance of the BQC task. If you need to fail the policy if coverage decreases, you have to compare it with the previous build. However, this setting does take a minimum threshold into account.

The only way to achieve your goal is by using two BQC tasks. The first should check the minimum threshold of 90% while the second should compare against the previous build.

[jrperron88-zz]

However, this setting does take a minimum threshold into account.

Can you elaborate on the use case? From my perspective, the code doesn't meet the minimum threshold so it should error. Just trying to understand how it actually works.

I will put in the two tasks to achieve this, just want to understand what the threshold controls

[ReneSchumacher]

This was a deliberate decision. Each policy in the BQC task has two different modes. You choose the mode by setting the *failOption parameter (e.g., https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/CodeCoveragePolicy.md#user-content-failoption).

In fixed mode the policy compares values with a fixed threshold, which was what people initially asked for. Fixed mode does not have any additional options since its purpose is to just make sure that, e.g., coverage never falls below a certain value.

Later we added build mode in which the policy compares values with those from the previous successful build. The idea behind that was that people don't want a defined threshold, which sometimes just leads to bad developer behavior. Instead, they want to make sure that coverage does not decrease once they reached a certain level. Once we had that, we received lots of customer suggestions about forcing improvements, variance, etc.

The upper threshold was added by us when we introduced the force improvement option, because it generally doesn't make much sense to strive for 100% coverage (unless you strictly do test-driven development and always are at 100%). Therefore, we added an option to tell the policy that further improvements are not required after you reached a certain upper threshold. Customers then asked for the ignore decrease above upper threshold option.

[jrperron88-zz]

Okay, I got it, your last paragraph explains what it is used for. Thank you for your help!