Issues on covering pull requests

OlavStornes commented 3 years ago

Greetings! When i attempt to set this up for checking code coverage in context of pull requests, I seem to have issues regarding No baseline found. In my case a branch with only 20 % covered is seen as valid when compared to the target branch, which has <70 % covered.

I've also added bqc/coverage-policy on the main branch as a status check, but this does not seem to trigger either. I have attempted to go through the documentation regarding pull requests and task parameters, but i still seem to have issues.

Is there something obvious i have missed?

Best regards, OlavStornes

Extra

Snippet of the PR-setup:

- task: BuildQualityChecks@8
  displayName: "Build quality check"
  continueOnError: ${{ parameters.ignore_failing_code_coverage_checks }}
  inputs:
    checkCoverage: true
    coverageFailOption: "build"
    coverageType: "branches"
    allowCoverageVariance: true
    coverageVariance: "1"
    coveragePrecision: "3"
    baseBranchRef: "refs/heads/main"

Log output:

##[debug]Evaluating condition for step: 'Build quality check'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Build quality check
==============================================================================
Task         : Build Quality Checks
Description  : Breaks a build based on quality metrics like number of warnings or code coverage.
Version      : 8.1.0
Author       : Microsoft
Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md)
==============================================================================
##[debug]Using node path: /home/vsts/agents/2.184.2/externals/node10/bin/node
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_CHECKWARNINGS
##[debug]loading INPUT_WARNINGFAILOPTION
##[debug]loading INPUT_WARNINGTHRESHOLD
##[debug]loading INPUT_FORCEFEWERWARNINGS
##[debug]loading INPUT_ALLOWWARNINGVARIANCE
##[debug]loading INPUT_SHOWSTATISTICS
##[debug]loading INPUT_EVALUATETASKWARNINGS
##[debug]loading INPUT_WARNINGTASKFILTERS
##[debug]loading INPUT_INCLUSIVEFILTERING
##[debug]loading INPUT_EVALUATEFILEWARNINGS
##[debug]loading INPUT_WARNINGFILESARTIFACT
##[debug]loading INPUT_CHECKCOVERAGE
##[debug]loading INPUT_COVERAGEFAILOPTION
##[debug]loading INPUT_COVERAGETYPE
##[debug]loading INPUT_TREAT0OF0AS100
##[debug]loading INPUT_COVERAGETHRESHOLD
##[debug]loading INPUT_FORCECOVERAGEIMPROVEMENT
##[debug]loading INPUT_COVERAGEUPPERTHRESHOLD
##[debug]loading INPUT_IGNOREDECREASEABOVEUPPERTHRESHOLD
##[debug]loading INPUT_USEUNCOVEREDELEMENTS
##[debug]loading INPUT_ALLOWCOVERAGEVARIANCE
##[debug]loading INPUT_COVERAGEVARIANCE
##[debug]loading INPUT_COVERAGEDELTATYPE
##[debug]loading INPUT_COVERAGEPRECISION
##[debug]loading INPUT_EXPLICITFILTER
##[debug]loading INPUT_INCLUDEPARTIALLYSUCCEEDED
##[debug]loading INPUT_FALLBACKONPRTARGETBRANCH
##[debug]loading INPUT_BASEDEFINITIONFILTER
##[debug]loading INPUT_BASEBRANCHREF
##[debug]loading INPUT_DISABLECERTCHECK
##[debug]loading INPUT_CREATEBUILDISSUES
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loading SECRET_C-GITHUB-REPLICATION-ACCESS-TOKEN
##[debug]loading SECRET_C-PULUMI-ACCESS-TOKEN
##[debug]loading SECRET_SANITY-DEPLOY-STUDIO-TOKEN
##[debug]loading SECRET_C-DEVOPS-BUILD-ACCESS-TOKEN
##[debug]loading SECRET_APP-MATCH-PASSWORD
##[debug]loaded 40
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]createBuildIssues=true
##[debug]check path : /home/vsts/work/_tasks/BuildQualityChecks_16a2ad20-f191-11e5-bed4-ab22bcd17937/8.1.0/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/BuildQualityChecks_16a2ad20-f191-11e5-bed4-ab22bcd17937/8.1.0/task.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/BuildQualityChecks_16a2ad20-f191-11e5-bed4-ab22bcd17937/8.1.0/node_modules/nodeapihelpers/messages.json
##[debug]adding resource file: /home/vsts/work/_tasks/BuildQualityChecks_16a2ad20-f191-11e5-bed4-ab22bcd17937/8.1.0/node_modules/nodeapihelpers/messages.json
##[debug]system.culture=en-US
##[debug]disableCertCheck=false
##[debug]SystemVssConnection exists true
##[debug]System.TeamFoundationCollectionUri={REDACTED_SITE_REPO}
##[debug]BQC.LogRawData=undefined
##[debug]Build.Repository.Provider={REDACTED}
##[debug]baseBranchRef=refs/heads/main
##[debug]baseDefinitionId=undefined
##[debug]System.DefinitionId=24
##[debug]includePartiallySucceeded=true
##[debug]fallbackOnPRTargetBranch=true
##[debug]System.TeamProject={REDACTED}
##[debug]System.DefinitionId=24
##[debug]Build.DefinitionVersion=2
##[debug]Build.BuildId=2289
##[debug]PSGer.Build.PollInterval=undefined
##[debug]PSGer.Build.MaxWaitTime=undefined
##[debug]Resource file has already set to: /home/vsts/work/_tasks/BuildQualityChecks_16a2ad20-f191-11e5-bed4-ab22bcd17937/8.1.0/node_modules/nodeapihelpers/messages.json
##[debug]System.PhaseDisplayName=Test phase
##[debug]System.PhaseName=test
##[debug]System.JobId={REDACTED}
##[debug]System.JobDisplayName=Test phase
Using IdentifierJobResolver
##[debug]Resource file has already set to: /home/vsts/work/_tasks/BuildQualityChecks_16a2ad20-f191-11e5-bed4-ab22bcd17937/8.1.0/node_modules/nodeapihelpers/messages.json
##[debug]Build.Repository.Provider={REDACTED}
##[debug]System.PullRequest.PullRequestId=375
##[debug]runTitle=undefined
##[debug]Build.Repository.ID={REPO_ID}
##[debug]System.TeamProject={REDACTED}
##[debug]System.CollectionUri={REDACTED_SITE_REPO}
##[debug]Build.BuildId=2289
##[debug]BQC.ForceNewBaseline=undefined
##[debug]getEnabledPolicies
##[debug]checkWarnings=false
##[debug]checkCoverage=true
##[debug] - coverage
##[debug]BQC.DisableVariableCreation=undefined
##[debug]getCurrentJobName
##[debug]runTitle=undefined
##[debug]waitForPreviousBuildStepsToFinish
##[debug]getPreviousBuildSteps
Validating code coverage policy...
##[debug]System.TeamProject={REDACTED}
##[debug]Build.BuildId=2289
##[debug]CoverageValueReader.getCoverageValue
##[debug]Build.BuildId=2289
##[debug]CoverageValueReader.getCodeCoverageSummaryFromBuild
##[debug]getCodeCoverageData
##[debug]prepareClients
Successfully read code coverage data from build.
##[debug]CoverageValueReader.filterCoverageDataByConfigAndPlatform
##[debug]buildConfiguration=undefined
##[debug]buildPlatform=undefined
##[debug]explicitFilter=false
##[debug]Inspecting 1 code coverage data sets...
##[debug]CoverageValueReader.aggregateCoverageValues
##[debug]coverageType=branches
Evaluating coverage data from 1 filtered code coverage data sets...
##[debug]CoverageValueReader.calculateCoveragePercentageWithPrecision
##[debug]coveragePrecision=3
##[debug]coverageVariance=1
Total branches: 760
Covered branches: 132
Code Coverage (%): 17.368
##[debug]coverageFailOption=build
##[debug]NewBaselineCoverageRule.isFulfilledImpl
##[debug]NewBaselineRule.evaluateRuleCore
##[debug]CompareToBaselinCoverageRule.isFulfilledImpl
##[debug]CompareToBaselinCoverageRule.getCoverageValueFromBaselineBuild
##[debug]getBaselineBuildId
##[debug]getBaselineBuild
##[debug]getCurrentJobStartTime
##[debug]Build.SourceBranch=refs/pull/375/merge
[WARNING] The baseline build could not be found. All policies based on the previous build will pass.
##[warning]The baseline build could not be found. All policies based on the previous build will pass.
##[debug]Processed: ##vso[task.logissue type=warning;]The baseline build could not be found. All policies based on the previous build will pass.
[SUCCESS] Code coverage policy passed with 17.368% (132/760 branches).
##[debug]set CodeCoveragePolicy.Result=passed
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Result;isOutput=false;issecret=false;]passed
##[debug]WarningsPolicy.createOutputVariables
##[debug]set CodeCoveragePolicyResult=passed
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicyResult;isOutput=false;issecret=false;]passed
##[debug]set CodeCoveragePolicy.Elements.Total=760
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Total;isOutput=false;issecret=false;]760
##[debug]set CodeCoveragePolicy.Elements.Covered=132
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Covered;isOutput=false;issecret=false;]132
##[debug]set CodeCoveragePolicy.Elements.Uncovered=628
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Uncovered;isOutput=false;issecret=false;]628
##[debug]set CodeCoveragePolicy.Percentage.Covered=17.368
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Percentage.Covered;isOutput=false;issecret=false;]17.368
##[debug]set CodeCoveragePolicy.Percentage.Uncovered=82.632
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Percentage.Uncovered;isOutput=false;issecret=false;]82.632
##[debug]set CodeCoveragePolicy.Elements.Label=branches
##[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Label;isOutput=false;issecret=false;]branches
##[debug]setTaskResult
##[debug]Agent.TempDirectory=/home/vsts/work/_temp
##[debug]Processed: ##vso[task.addattachment type=Distributedtask.Core.Summary;name=Build Quality Checks;]/home/vsts/work/_temp/BuildQualityChecksSummary.md
##[section]Finishing: Build quality check

ReneSchumacher commented 3 years ago

Hi Olav,

when running as part of a pull request validation build, BQC uses the following logic if configured to compare with the previous build:

It looks for a succeeded build for the exact same branch first. For PRs that branch is something like refs/pull/{PR ID}/merge. Naturally, it won't find a build for the very first build run for a PR.
If there is no previous build and BQC detects the PR context (by looking at the System.PullRequest.TargetBranch variable), it automatically tries to find a suceeded build that ran for the PR target branch and uses that as a baseline.
If you specify a specific baseBranchRef that doesn't match the source branch, BQC looks for a succeeded build that ran for the given branch.

According to your log, BQC neither fell back on the PR target branch (there would have been a log message for that), nor did it find a baseline build. Are you sure that

the same build definition you are using for your PR validation did run for the main branch, and
the build result was succeeded and not partially succeeded?

If you are using different build definitions for PR builds and CI builds, you also have to specify the baseDefinitionId parameter.

OlavStornes commented 3 years ago

Hi Rene,

I am not sure if the same build definition was used, which i assume is the culprit in my case.
It seems as the build results did succeed fully.

After digging around, i figured i had multiple pipeline builds, which threw me off guard. The solution was to set baseDefinitionId to my projects pipeline build ID when running a PR. Now it fails successfully!

Thank you for replying and your assistance.

MicrosoftPremier / VstsExtensions