Warnings not reported, and task filter not match build task

[atdheekurteshi]

Here us build log excerpt, full attached: ([WARNING] The specified task filters did not match any build task)

[debug]coverageType=lines

Evaluating coverage data from 1 filtered code coverage data sets...

[debug]CoverageValueReader.calculateCoveragePercentageWithPrecision

[debug]coveragePrecision=4

[debug]coverageVariance=undefined

Total lines: 3 Covered lines: 2 Code Coverage (%): 66.6667

[debug]coverageFailOption=fixed

[debug]FixedCoverageThresholdRule.isFulfilledImpl

[debug]coverageThreshold=90

[ERROR] The code coverage value (66.6667%, 2 lines) is lower than the minimum value (90%)!

[error]The code coverage value (66.6667%, 2 lines) is lower than the minimum value (90%)!

[debug]Processed: ##vso[task.logissue type=error;]The code coverage value (66.6667%, 2 lines) is lower than the minimum value (90%)!

[debug]set CodeCoveragePolicy.Result=failed

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Result;isOutput=false;issecret=false;]failed

[debug]CodeCoveragePolicy.createOutputVariables

[debug]set CodeCoveragePolicyResult=failed

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicyResult;isOutput=false;issecret=false;]failed

[debug]set CodeCoveragePolicy.Elements.Total=3

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Total;isOutput=false;issecret=false;]3

[debug]set CodeCoveragePolicy.Elements.Covered=2

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Covered;isOutput=false;issecret=false;]2

[debug]set CodeCoveragePolicy.Elements.Uncovered=1

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Uncovered;isOutput=false;issecret=false;]1

[debug]set CodeCoveragePolicy.Percentage.Covered=66.6667

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Percentage.Covered;isOutput=false;issecret=false;]66.6667

[debug]set CodeCoveragePolicy.Percentage.Uncovered=33.3333

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Percentage.Uncovered;isOutput=false;issecret=false;]33.3333

[debug]set CodeCoveragePolicy.Elements.Label=lines

[debug]Processed: ##vso[task.setvariable variable=CodeCoveragePolicy.Elements.Label;isOutput=false;issecret=false;]lines

[debug]setTaskResult

[debug]Agent.TempDirectory=D:\a_temp

[debug]Processed: ##vso[task.addattachment type=Distributedtask.Core.Summary;name=Build Quality Checks;]D:\a_temp\BuildQualityChecksSummary.md

[debug]createBuildIssues=true

[debug]task result: Failed

[debug]Processed: ##vso[task.complete result=Failed;]

Finishing: Check build quality

Here is my task, full yml attached:

trigger:

• main
• rel/* pr:
• main
• rel/*

variables: DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1 CI: 1 workingDirectory: '$(System.DefaultWorkingDirectory)/' buildPlatform: 'Any CPU' buildConfiguration: 'Release'

stages:

• stage: Build jobs:

  • job: build workspace: clean: all pool: vmImage: windows-2022 timeoutInMinutes: 3 steps:
    • checkout: self fetchDepth: 1
    • powershell: ./scripts/Build.ps1
    • publish: $(Build.SourcesDirectory) artifact: all

• stage: Test jobs:

  • job: 'CodeCoverage' workspace: clean: all strategy: matrix: PS7_Windows_Server2022: vmImage: windows-2022 pwsh: true pool: vmImage: $[ variables['vmImage'] ] steps:
    • powershell: ./scripts/CodeCoverage.ps1
    • task: PublishCodeCoverageResults@1 inputs: codeCoverageTool: 'JaCoCo' summaryFileLocation: 'Pester-Coverage.xml' pathToSources: '$(Build.SourcesDirectory)' failIfCoverageEmpty: false condition: succeededOrFailed()
    • task: PublishTestResults@2 inputs: testResultsFormat: 'NUnit' testResultsFiles: 'Test-Results.xml' failTaskOnFailedTests: true condition: succeededOrFailed() timeoutInMinutes: 20

• stage: QA dependsOn: Test jobs:

  • job: 'BuildQualityChecks' workspace: clean: all strategy: matrix: PS7_Windows_Server2022: vmImage: windows-2022 pwsh: true pool: vmImage: $[ variables['vmImage'] ] steps:
    • task: BuildQualityChecks@8 displayName: 'Check build quality' inputs:

      ===== Warnings Policy Inputs =====

      checkWarnings: false # Optional warningFailOption: build # Optional; Valid values: build, fixed warningThreshold: '20' # Optional forceFewerWarnings: false # Optional allowWarningVariance: false # Optional

      warningVariance: # Required if allowWarningVariance = true

      showStatistics: true # Optional evaluateTaskWarnings: true # Optional

      warningTaskFilters: '/^((vs|ms)build|ant(\s+.+)?|gradle(w)?(\s+.+)?|grunt|gulp|maven(\s+.+)?|xamarin(android|ios)|xcode(\s+.+)?|cmake|build\s+.+)$/i' # Optional

      warningFilters: # Optional

      inclusiveFiltering: false # Optional

      evaluateFileWarnings: false # Optional

      warningFilesFolder: # Optional

      warningFiles: # Required if evaluateFileWarnings = true

      warningFileFilters: # Required if evaluateFileWarnings = true

      warningFilesArtifact: # Required if evaluateFileWarnings = true and (warningFailOption = build or showStatistics = true)

      checkCoverage: true coverageFailOption: 'fixed' coverageType: 'lines' coverageThreshold: '20' includePartiallySucceeded: true fallbackOnPRTargetBranch: true baseDefinitionFilter: '*' baseDefinitionId: 'PSCode' baseBranchRef: '$(System.PullRequest.TargetBranch)' timeoutInMinutes: 20

---

Raw Data File https://www.transfernow.net/dl/Desktop

[ReneSchumacher]

Hi @atdheekurteshi,

can you tell me what you are trying to achieve in your pipeline? Your title says that warnings are not reported and according to your yaml definition, you have disabled the warnings policy (checkWarnings: false). Thus, the task should not even try to evaluate warnings. I also cannot find an attachment with the full pipeline definition. You can send it over via email to PSGerExtSupport@microsoft.com.

Your title also mentions that the task filters don't match. I'm wondering why you see that message when the warnings policy is disabled, so I'm guessing that the yaml you posted is not exactly what's in your pipeline definition. If I assume that checkWarnings is indeed set to true, the reason for warnings not being reported is that you put the BuildQualityChecks task into a separate job. The task can only evaluate warnings from the job in which it is executed. In addition, it needs to be able to inspect the build timeline and log information from the pipeline tasks. It does this automatically for most commonly used build tasks. However, you seem to have put all your build and test logic into PowerShell scripts (Build.ps1 and CodeCoverage.ps1), which our task does not understand by default.

Could you send me (email to PSGerExtSupport@microsoft.com)

• the full yaml definition of your pipeline
• the full log output from your pipeline

With these files I should be able to tell you how to configure BQC for your scenario.

[atdheekurteshi]

Thanks, @ReneSchumacher I have sent it.

[atdheekurteshi]

Thanks, @ReneSchumacher I have sent it.

[atdheekurteshi]

Hi Rene!

Yes, the scenario of my YAML is to show the code coverage of my PowerShell pester test files and to output a warning if the threshold is below 90% but not stop the pipeline to perform only to warn the user that there is a warning and your code-coverage or build quality checks are below 90% threshold.

CodeCoverage is generated from JaCoCo which I used in my YAML file, and it's working fine, the only issue that is not working with BuildQualityChecks is not showing Warnings.

Below, you have attached the:

• the full YAML definition of your pipeline
• the full log output from your pipeline

regarding this issue: https://github.com/MicrosoftPremier/VstsExtensions/issues/181 BR, Atdhe Kurteshi

On Tue, 19 Apr 2022 at 12:45, René Schumacher @.***> wrote:

Hi @atdheekurteshi https://github.com/atdheekurteshi,

can you tell me what you are trying to achieve in your pipeline? Your title says that warnings are not reported and according to your yaml definition, you have disabled the warnings policy (checkWarnings: false). Thus, the task should not even try to evaluate warnings. I also cannot find an attachment with the full pipeline definition. You can send it over via email to @.***

Your title also mentions that the task filters don't match. I'm wondering why you see that message when the warnings policy is disabled, so I'm guessing that the yaml you posted is not exactly what's in your pipeline definition. If I assume that checkWarnings is indeed set to true, the reason for warnings not being reported is that you put the BuildQualityChecks task into a separate job. The task can only evaluate warnings from the job in which it is executed. In addition, it needs to be able to inspect the build timeline and log information from the pipeline tasks. It does this automatically for most commonly used build tasks. However, you seem to have put all your build and test logic into PowerShell scripts (Build.ps1 and CodeCoverage.ps1), which our task does not understand by default.

Could you send me (email to @.***)

• the full yaml definition of your pipeline
• the full log output from your pipeline

With these files I should be able to tell you how to configure BQC for your scenario.

— Reply to this email directly, view it on GitHub https://github.com/MicrosoftPremier/VstsExtensions/issues/181#issuecomment-1102493958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANZD7VUGP72KP7NPZX2MJTVF2FFHANCNFSM5TYKW4PA . You are receiving this because you were mentioned.Message ID: @.***>

trigger:

• main
• rel/* pr:
• main
• rel/*

variables: DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1 CI: 1 workingDirectory: '$(System.DefaultWorkingDirectory)' buildPlatform: 'Any CPU' buildConfiguration: 'Release'

stages:

• stage: Build jobs:
  • job: build workspace: clean: all condition: succeededOrFailed() strategy: matrix: PS7_Windows_Server2022: vmImage: windows-2022 continueOnError: true pool: vmImage: windows-2022 timeoutInMinutes: 3 steps:
    • powershell: Install-Module -Name 'InjectionHunter' -Scope CurrentUser -Force | Import-Module -Name 'InjectionHunter'
    • checkout: self fetchDepth: 1
    • powershell: ./scripts/Build.ps1
    • publish: $(Build.SourcesDirectory) artifact: all

• stage: Test dependsOn: Build jobs:

  • job: 'CodeCoverage' workspace: clean: all condition: succeededOrFailed() strategy: matrix: PS7_Windows_Server2022: vmImage: windows-2022 pwsh: true continueOnError: true pool: vmImage: $[ variables['vmImage'] ] steps:

    • powershell: ./scripts/CodeCoverage.ps1
    • task: @.*** inputs: codeCoverageTool: 'JaCoCo' summaryFileLocation: 'Pester-Coverage.xml' pathToSources: '$(System.DefaultWorkingDirectory)/tests' failIfCoverageEmpty: true condition: succeededOrFailed()
    • task: @.*** inputs: testResultsFormat: 'NUnit' testResultsFiles: 'Test-Results.xml' failTaskOnFailedTests: true condition: succeededOrFailed() timeoutInMinutes: 20

  • job: 'BuildQualityChecks' workspace: clean: all dependsOn: 'CodeCoverage' condition: succeededOrFailed() strategy: matrix: PS7_Windows_Server2022: vmImage: windows-2022 pwsh: true pool: vmImage: $[ variables['vmImage'] ] steps:

    • task: @.*** displayName: 'Check build quality' inputs:

      ===== Warnings Policy Inputs =====

      checkWarnings: true # Optional warningFailOption: 'fixed' # Optional; Valid values: build, fixed warningThreshold: '90' # Optional

      createBuildIssues: true

      forceFewerWarnings: false # Optional

      allowWarningVariance: true # Optional

      warningVariance: '1' # Required if allowWarningVariance = true''

      showStatistics: true # Optional

      evaluateTaskWarnings: true # Optional

      warningTaskFilters: '/^((vs|ms)build|ant(\s+.+)?|gradle(w)?(\s+.+)?|grunt|gulp|maven(\s+.+)?|xamarin(android|ios)|xcode(\s+.+)?|cmake|build\s+.+)$/i' # Optional

      warningFilters: # Optional

      inclusiveFiltering: false # Optional

      evaluateFileWarnings: true # Optional

      warningFilesFolder: '$(Build.SourcesDirectory)'

      warningFiles: # Required if evaluateFileWarnings = true

      warningFileFilters: # Required if evaluateFileWarnings = true

      warningFilesArtifact: # Required if evaluateFileWarnings = true and (warningFailOption = build or showStatistics = true)

      checkCoverage: true coverageFailOption: 'fixed' coverageType: 'lines' coverageThreshold: '90' includePartiallySucceeded: true fallbackOnPRTargetBranch: true buildConfiguration: '$(BuildPlatform)' baseDefinitionFilter: '*' baseDefinitionId: 'PSCode' baseBranchRef: '$(System.PullRequest.TargetBranch)' timeoutInMinutes: 20

[ReneSchumacher]

Hi @atdheekurteshi,

as I understand it, you are only interested in a fixed code coverage threshold of 90% and just want to see a "warning" in your pipeline if coverage is below that value. The correct YAML snippet for this configuration looks like this:

- task: BuildQualityChecks@8
  displayName: 'Check build quality'
  continueOnError: true # do not break the pipeline if the task fails
  timeoutInMinutes: 20 # may not be necessary, the task times out internally after 10 minutes (default)
  inputs:
    checkCoverage: true # enable the coverage policy
    coverageFailOption: fixed # set the policy to fixed threshold mode
    coverageType: lines # look at line coverage data
    coverageThreshold: 90 # set a fixed threshold of 90%

The important bits are:

• Do not enable the warnings policy since you are not interested in compiler or other warnings from pipeline tasks.
• Set continueOnError to true. The task does not support issuing a warning, it always fails if a policy did not pass. Setting continueOnError: true marks the task as failed and your pipeline as partially succeeded.

[atdheekurteshi]

@ReneSchumacher Fine it works but I hope in the near future you will implement also the Warning Issues.

[ReneSchumacher]

Hi again,

thanks for the feedback. I don't think we're going to have an option to reduce the task outcome to just a warning. If you think of the task as a policy, then the only two outcomes are either passed or failed. And using the continueOnError property is in line with most other tasks and policies in Azure DevOps (e.g., pull request policies). I don't think there would be a benefit to changing that behavior.

Since the issue is resolve, I'm closing it now. However, please feel free to add additional comments for further feedback or open a new issue if you need additional help.