Warnings not detected in current build

[mjvankampen]

Running build quality checks with warning task filters seem to match for builds that have been done (baseline) but not for the current build. How is this possible? It depends on other jobs, i.e.:

dependsOn:
      - BuildAndTestMasterCoverage
      - StaticAnalysisMaster

Log:

Starting: Check build quality against previous commit
==============================================================================
Task         : Build Quality Checks
Description  : Breaks a build based on quality metrics like number of warnings or code coverage.
Version      : 6.3.0
Author       : Microsoft Premier Services
Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md)
==============================================================================
SystemVssConnection exists true
Using IdentifierJobResolver
Validating build warnings policy...
[WARNING] The specified task filters did not match any build task.
##[warning]The specified task filters did not match any build task.
Total warnings: 0
Filtered warnings: 0
Found baseline build with ID 9870.
Counting warnings from tasks:
- Run linters
-   Total warnings: 0, Filtered warnings: 0
- Run linters
-   Total warnings: 114, Filtered warnings: 114
Total warnings: 114
Filtered warnings: 114
[SUCCESS] Warnings policy passed with 0 warning(s).
Validating code coverage policy...
Waiting for code coverage data...
Waiting for code coverage data...
Waiting for code coverage data...
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total branches: 992
Covered branches: 454
Code Coverage (%): 45.7661
Found baseline build with ID 9870.
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total branches: 992
Covered branches: 454
Code Coverage (%): 45.7661
[SUCCESS] Code coverage policy passed with 45.7661% (454/992 branches).
Finishing: Check build quality against previous commit

yaml:

- job: CheckQualityMaster
        displayName: Check MASTER quality
        steps:
          - checkout: none
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against previous commit"
            condition: not(eq(variables['Build.Reason'], 'PullRequest'))
            inputs:
              checkWarnings: true
              warningFailOption: build
              warningTaskFilters: /.*lint.*/i
              checkCoverage: true
              coverageFailOption: build
              coverageType: branches
              coverageDeltaType: absolute
              forceCoverageImprovement: false
              allowCoverageVariance: false
              explicitFilter: true
              useUncoveredElements: true

[ReneSchumacher]

Hey,

thank you for the feedback. Could you send me the full YAML file? I don't think we currently support checking warnings from different build jobs. If the task that creates warnings resides in a job other than the one the BQC task is in, it won't find them. It shouldn't find the warnings from the previous build if they were created in a different job.

Once I know the layout of your build I can try to reproduce and think about a solution. The challenge with cross-job evaluation is that we need to ensure to compare correlating jobs. Otherwise you might be comparing totally unrelated warnings.

René

[mjvankampen]

Hi Rene, somehow my post got lost. Anyway I now run all build quality tasks in the job they use data from. Works like a charm! Not sure why coverage used to work though :-).

My YAML looks like this, where before the build quality task was in its own job:

trigger:
  branches:
    include:
      - "*"
  paths:
    include:
      - MASTER/*
      - COMMON/*
      - .azuredevops/*
      - azure-master-pipeline.yml
stages:
  - stage: BuildAndTestMasterRelease
    displayName: MASTER Release
    dependsOn: []
    jobs:
      - job: BuildAndTestMasterQnx
        pool:
          name: Default
          demands:  
          - docker
          - agent.os -equals Windows_NT
        container: windows
        workspace:
          clean: outputs
        displayName: Build and test MASTER for QNX
        steps:
          - template: ".azuredevops/azure-build-template.yml"
            parameters:
              buildConfiguration: Release
              buildProfile: qnx_profile
              sourceSubfolder: MASTER
          - script: $(Build.BinariesDirectory)/activate && ctest -T Test --no-compress-output --output-on-failure -j 5
            displayName: "CTest"
            workingDirectory: $(Build.BinariesDirectory)
            failOnStderr: false
          - task: PublishTestResults@2
            displayName: "Publish test results"
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: "CTest"
              testResultsFiles: "**/Test.xml"
              testRunTitle: "MASTER Release"
              buildConfiguration: "Release"
              searchFolder: "$(Build.BinariesDirectory)/Testing"
              mergeTestResults: true
          - task: PublishBuildArtifacts@1
            displayName: "Publish artifacts"
            inputs:
              PathtoPublish: $(Build.ArtifactStagingDirectory)
              ArtifactName: "artifacts_$(Build.SourceBranchName)_Release_$(Build.BuildNumber)"
              publishLocation: "Container"

  - stage: BuildAndTestMasterCoverage
    dependsOn: []
    displayName: MASTER Coverage
    jobs:
      - job: BuildAndTestMasterQnx
        pool:
          name: Default
          demands:  
          - docker
          - agent.os -equals Windows_NT
        container: windows
        workspace:
          clean: outputs
        displayName: Build and test MASTER for QNX
        steps:
          - template: ".azuredevops/azure-build-template.yml"
            parameters:
              buildConfiguration: Coverage
              buildProfile: qnx_profile
              sourceSubfolder: MASTER
          - script: $(Build.BinariesDirectory)/activate && ctest -T Test --no-compress-output --output-on-failure -j 5
            displayName: "CTest"
            workingDirectory: $(Build.BinariesDirectory)
            failOnStderr: false
          - task: PublishTestResults@2
            displayName: "Publish test results"
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: "CTest"
              testResultsFiles: "**/Test.xml"
              searchFolder: "$(Build.BinariesDirectory)/Testing"
              mergeTestResults: true
              testRunTitle: "MASTER Coverage"
              buildConfiguration: "Coverage"
          - task: PublishCodeCoverageResults@1
            displayName: "Publish code coverage report"
            inputs:
              codeCoverageTool: "Cobertura"
              summaryFileLocation: "$(Build.BinariesDirectory)/TEST_COVERAGE.xml"
              pathToSources: "."
              reportDirectory: "$(Build.BinariesDirectory)/code_coverage"
              failIfCoverageEmpty: true
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against previous commit"
            condition: not(eq(variables['Build.Reason'], 'PullRequest'))
            inputs:
              checkCoverage: true
              coverageFailOption: build
              coverageType: branches
              coverageDeltaType: absolute
              forceCoverageImprovement: false
              allowCoverageVariance: false
              explicitFilter: true
              useUncoveredElements: true
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against target branch"
            condition: eq(variables['Build.Reason'], 'PullRequest')
            inputs:
              baseBranchRef: $(System.PullRequest.TargetBranch)
              checkCoverage: true
              coverageFailOption: build
              coverageType: branches
              coverageDeltaType: absolute
              forceCoverageImprovement: false
              allowCoverageVariance: false
              explicitFilter: true
              useUncoveredElements: true

  - stage: BuildAndTestMasterDebug
    displayName: MASTER Debug
    dependsOn: []
    jobs:
      - job: BuildAndTestMasterQnx
        pool:
          name: Default
          demands:  
          - docker
          - agent.os -equals Windows_NT
        container: windows
        workspace:
          clean: outputs
        displayName: Build and test MASTER for QNX
        steps:
          - template: ".azuredevops/azure-build-template.yml"
            parameters:
              buildConfiguration: Debug
              buildProfile: qnx_profile
              sourceSubfolder: MASTER
          - script: $(Build.BinariesDirectory)/activate && ctest -T Test --no-compress-output --output-on-failure -j 5
            displayName: "CTest"
            failOnStderr: false
            workingDirectory: $(Build.BinariesDirectory)
          - task: PublishTestResults@2
            displayName: "Publish test results"
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: "CTest"
              testResultsFiles: "**/Test.xml"
              testRunTitle: "MASTER Debug"
              buildConfiguration: "Debug"
              searchFolder: "$(Build.BinariesDirectory)/Testing"
              mergeTestResults: true

  - stage: StaticAnalysisMaster
    displayName: "Static analysis of the MASTER"
    dependsOn: []
    jobs:
      - job: Cppcheck
        pool:
          name: Default
          demands:  
          - docker
          - agent.os -equals Windows_NT
        container: windows_vscpp
        workspace:
          clean: outputs
        displayName: Lint MASTER with Cppcheck
        steps:
          - template: ".azuredevops/azure-conan-template.yml"
            parameters:
              buildConfiguration: Release
              buildProfile: qnx_profile
              sourceSubfolder: MASTER
          - script: $(Build.BinariesDirectory)/activate && cmake --build $(Build.BinariesDirectory) --target LINT_WITH_CPPCHECK -j 10
            displayName: "Run cpp check"
            workingDirectory: MASTER
            failOnStderr: false
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against previous commit"
            condition: not(eq(variables['Build.Reason'], 'PullRequest'))
            inputs:
              checkWarnings: true
              warningFailOption: build
              warningTaskFilters: /.*Run cpp check.*/i
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against target branch"
            condition: eq(variables['Build.Reason'], 'PullRequest')
            inputs:
              baseBranchRef: $(System.PullRequest.TargetBranch)
              checkWarnings: true
              warningFailOption: build
              warningTaskFilters: /Run cpp check/i
      - job: PClint_AUTOSAR
        pool:
          name: Default
          demands:  
          - docker
          - agent.os -equals Windows_NT
        container: windows
        workspace:
          clean: outputs
        displayName: Lint MASTER with PC-lint for AUTOSAR
        steps:
          - template: ".azuredevops/azure-conan-template.yml"
            parameters:
              buildConfiguration: Release
              buildProfile: qnx_profile
              sourceSubfolder: MASTER
          - script: $(Build.BinariesDirectory)/activate && cmake --build $(Build.BinariesDirectory) --target LINT_WITH_PC_LINT -j 10
            displayName: "Run PC-lint"
            workingDirectory: MASTER
            failOnStderr: false
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against previous commit"
            condition: not(eq(variables['Build.Reason'], 'PullRequest'))
            inputs:
              checkWarnings: true
              warningFailOption: build
              warningTaskFilters: /Run PC-lint/i
          - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
            displayName: "Check build quality against target branch"
            condition: eq(variables['Build.Reason'], 'PullRequest')
            inputs:
              baseBranchRef: $(System.PullRequest.TargetBranch)
              checkWarnings: true
              warningFailOption: build
              warningTaskFilters: /Run PC-lint/i

[ReneSchumacher]

Thanks for confirming that it's working! :)

Coverage is a different story because coverage data is not tied to a particular build job but to the overall build. ;-) So as long as you make sure that the coverage data is published before the BQC task runs, it'll find it.