Template Analyzer - Template / Validation Errors Cause Failure, no output to Sarif Scan Tab

[rmazurik]

When working with the MSDO tool in our IaC Build Pipeline, we're experiencing an issue with our template : Error exit code 22: 22. Violation + Error

That is fine and understandable, but what's troubling is we're not able to view scan results in the Sarif Scan Tab, it seems the sarif files never get the chance to get loaded into the CodeAnalysisLogs artifact when this failure occurs. I'm wondering if this may be addressed in the code to allow for more meaningful error messages / remediation steps for these types of validation / error messages. to be displayed in this scans tab. Currently I'm having to sift through ~60,000 lines to find the error occurrence and it's hard to track down where in our files the failure is occurring.

[sivabalan-ammaiyappan]

I am also facing similar issue on the Build pipeline. Error running templateanalyzer job: 1 of 1 GuardianErrorExitCodeException: templateanalyzer completed with an Error exit code: 22. Violation + Error: Scan encountered both violations in template(s) and errors trying to analyze template(s)

[bjompen]

Same here with Binskim. I have an exe file, licecap.exe from cockos, in my test / demo setup, and it fails binskim with error

D:\a\1\s\DFDOFiles\licecap.exe : error ERR997.ExceptionLoadingPdb : 'licecap.exe' was not evaluated because its PDB could not be loaded (E_PDB_NO_DEBUG_INFO).

## <cut for brevity>
    ------------------------------------------------------------------------------
    BinSkim completed with exit code 1
##[error]Error running binskim job: 1 of 1
##[error]GuardianErrorExitCodeException: binskim completed with an Error exit code: 1. BinSkim failed. Verify the target(s) to be scanned. BinSkim targets must be a specific filename, or a pattern with a wildcard like *.dll, dir\*.dll, or dir\*

Adding an exclusion in .gdnconfig like this

{
  "tools": [
    {
      "tool": {
        "name": "BinSkim",
        "version": "Latest"
      },
      "arguments": {
        "IgnorePdbLoadError": "true"
      }
    }
  ]
}

fixes the error and .sarif output is created

The tests do continue, and my log also has errors for other demo /test files, but the sarif output stops as soon as an error is output to the log.

[bjompen]

A wild guess would be, judging from my binskim error and looking at the source of the extension, that terminating errors from the runtime itself and not errors in the scanned files, throws and stops the task who then doesn't create the .sarif files.

One potential workaround might be to turn of the sarif output by setting the inputs -> publish value to false and then manually publishing it with a condition of always() like this

- task: MicrosoftSecurityDevOps@1
  displayName: 'Microsoft Security DevOps'
  inputs:
    config: 'gdnconfig/.gdnconfig'
    publish: false

- task: PublishPipelineArtifact@1
  condition: always()
  inputs:
    targetPath: '$(System.DefaultWorkingDirectory)\.gdn\.r'
    artifact: 'CodeAnalysisLogs'
    publishLocation: 'pipeline'

This at least makes any sarif files available, however it seems like if one step fails then no sarif files exists even if other steps complete.

A better solution might be to have the extension compiling and publishing the sarif file outside of the run() function in the task, but I don't know how or even if that is possible..

[boAndron]

Fyi; a fix for Template Analyzer was deployed to deal with that issue. Also, a fix was deployed to prevent breaking the whole flow if a tool does not execute successfully. The exception will be caught and re-thrown at the end, after post-processing has completed.