BinSkim SARIF log parsing issue (v.1.9.1)

teemukj commented 9 months ago

In the 1.9.1 release of the MicrosoftSecurityDevOps extension, there seems to be an issue with the SARIF log (or the parsing). The extension is failing with JsonReaderException: Unexpected character encountered while parsing value: S. Path '', line 0, position 0. error. The scan is executed with MS hosted Azure DevOps (Windows) pipeline agent.

With some testing with the same repository/filebase and identical scan configuration, this seems to happen when only the BinSkim log is found and parsed. With previous version 1.9.0, the initial scan results are the the same (all tools exiting with code 0), but the final parsing step finds a Trivy log as well as the BinSkim log, and the parsing succeeds. Example log of the parsing with 1.9.0:

Process:
    Convert:
      Converting any raw tool logs to Sarif format ...
      Found 1 logs for tool trivy.
      Found 1 logs for tool binskim.
      Completed converting raw tool logs to Sarif format.

Where as this is what happens with identical scan with version 1.9.1:

Process:
    Convert:
      Converting any raw tool logs to Sarif format ...
      Found 1 logs for tool binskim.
      Completed converting raw tool logs to Sarif format.
    Import:
##[error]JsonReaderException: Unexpected character encountered while parsing value: S. Path '', line 0, position 0.
##[error]MSDO CLI exited with an error exit code: 1

This effectively fails the whole run, and no scan output log artifact is produced at all.

Scan configuration file:

```json { "tools": [ { "tool": { "name": "Trivy", "version": "Latest" }, "arguments": { "Target": "$(WorkingDirectory)", "Action": "filesystem" } }, { "tool": { "name": "CredScan", "version": "Latest" }, "arguments": { "TargetDirectory": "$(WorkingDirectory)", "OutputType": "sarif" } }, { "tool": { "name": "BinSkim", "version": "Latest" }, "arguments": { "Target": "$(WorkingDirectory)/*", "SarifOutputVersion": "Current", "Function": "analyze", "Recurse": "true" } }, { "tool": { "name": "AntiMalware", "version": "Latest" }, "arguments": { "ScanDirectoryOrFile": "$(WorkingDirectory)", "Function": "analyze", "Command": "scan", "ScanType": 3 } }, { "tool": { "name": "Bandit", "version": "Latest" }, "arguments": { "Target": "$(WorkingDirectory)", "Format": "sarif", "Recursive": "" } } ] } ```

Redacted full log output from a failed run:

```console 2023-10-13T11:11:17.3703792Z ##[section]Starting: Microsoft Security for DevOps 2023-10-13T11:11:17.3811223Z ============================================================================== 2023-10-13T11:11:17.3811551Z Task : Microsoft Security DevOps 2023-10-13T11:11:17.3811754Z Description : Run the Microsoft Security DevOps CLI for static analysis. 2023-10-13T11:11:17.3812149Z Version : 1.9.1 2023-10-13T11:11:17.3812250Z Author : Microsoft Corporation 2023-10-13T11:11:17.3812434Z Help : Runs the [Microsoft Security DevOps CLI](https://aka.ms/msdo-nuget) for security analysis. Effective September 20th 2023, the Secret Scanning option (CredScan) within Microsoft Security DevOps (MSDO) Extension for Azure DevOps is deprecated. MSDO Secret Scanning is replaced by the [Configure GitHub Advanced Security for Azure DevOps features](https://learn.microsoft.com/en-us/azure/devops/repos/security/configure-github-advanced-security-features#set-up-secret-scanning) offering. 2023-10-13T11:11:17.3815536Z ============================================================================== 2023-10-13T11:11:19.3647063Z ------------------------------------------------------------------------------ 2023-10-13T11:11:19.3651116Z Installing Microsoft Security DevOps Cli version: Latest 2023-10-13T11:11:22.7788185Z Installed Microsoft.Security.Devops.Cli.win-x64 version 0.182.0 2023-10-13T11:11:22.7790434Z ------------------------------------------------------------------------------ 2023-10-13T11:11:22.7815286Z [command]C:\Windows\system32\cmd.exe /D /S /C "D:\a\_msdo\versions\Microsoft.Security.Devops.Cli.win-x64.0.182.0\tools\guardian.cmd init --force" 2023-10-13T11:11:24.7640605Z [command]C:\Windows\system32\cmd.exe /D /S /C "D:\a\_msdo\versions\Microsoft.Security.Devops.Cli.win-x64.0.182.0\tools\guardian.cmd run -c D:\a\1\s\pipeline\security\.gdnconfig -p azuredevops --rich-exit-code --logger-pipeline --export-breaking-results-to-file D:\a\1\a\.gdn\msdo.sarif --telemetry-environment azdevops" 2023-10-13T11:11:25.7963002Z Run: 2023-10-13T11:11:25.9371492Z Installing Microsoft.Security.CodeAnalysis.Policy.Names 2023-10-13T11:11:27.6689413Z Verified package: Microsoft.Security.CodeAnalysis.Policy.Names 1.0.3 2023-10-13T11:11:27.6695327Z ------------------------------------------------------------------------------ 2023-10-13T11:11:27.6809317Z Installing Microsoft.Security.CodeAnalysis.Policy.AzureDevOps 2023-10-13T11:11:28.3702401Z Verified package: Microsoft.Security.CodeAnalysis.Policy.AzureDevOps 1.0.0 2023-10-13T11:11:28.3703482Z ------------------------------------------------------------------------------ 2023-10-13T11:11:28.6454254Z Install: 2023-10-13T11:11:28.6957555Z Installing Microsoft.Guardian.TrivyRedist_windows_amd64 2023-10-13T11:11:32.7055250Z Verified package: Microsoft.Guardian.TrivyRedist_windows_amd64 0.45.0 2023-10-13T11:11:32.7056750Z ------------------------------------------------------------------------------ 2023-10-13T11:11:32.7064469Z Installing Microsoft.Security.CredScan 2023-10-13T11:12:24.0221804Z Verified package: Microsoft.Security.CredScan 2.5.1.13 2023-10-13T11:12:24.0230276Z ------------------------------------------------------------------------------ 2023-10-13T11:12:24.0232897Z Installing Microsoft.CodeAnalysis.BinSkim 2023-10-13T11:13:02.3005710Z Verified package: Microsoft.CodeAnalysis.BinSkim 1.9.5 2023-10-13T11:13:02.3007121Z ------------------------------------------------------------------------------ 2023-10-13T11:13:02.3029341Z Installing Microsoft.Security.DevOps.AntiMalware.Cli.win-x64 2023-10-13T11:13:17.6921351Z Verified package: Microsoft.Security.DevOps.AntiMalware.Cli.win-x64 1.7.0 2023-10-13T11:13:17.6925228Z ------------------------------------------------------------------------------ 2023-10-13T11:13:17.7121947Z Installing Microsoft.Guardian.BanditRedist_windows_amd64 2023-10-13T11:13:34.3249181Z Successfully installed 'Microsoft.Guardian.BanditRedist_windows_amd64 1.6.3.1' to D:\a\_msdo\packages\nuget 2023-10-13T11:13:34.3252294Z ------------------------------------------------------------------------------ 2023-10-13T11:13:34.3272960Z Analyze: 2023-10-13T11:13:34.3652873Z Running Trivy 0.45.0 2023-10-13T11:13:39.9839770Z Trivy completed with exit code 0 2023-10-13T11:13:40.0010842Z ------------------------------------------------------------------------------ 2023-10-13T11:13:40.0011588Z 2023-10-13T11:13:40.0063307Z Running Credential Scanner 2.5.1.13 2023-10-13T11:14:02.6854684Z Credential Scanner completed with exit code 0 2023-10-13T11:14:02.6855912Z ------------------------------------------------------------------------------ 2023-10-13T11:14:02.6856443Z 2023-10-13T11:14:02.6982066Z Running BinSkim 1.9.5 2023-10-13T11:14:02.6982888Z ------------------------------------------------------------------------------ 2023-10-13T11:14:02.7028811Z D:\a\_msdo\packages\nuget\Microsoft.CodeAnalysis.BinSkim.1.9.5\tools\netcoreapp3.1\win-x64\BinSkim.exe analyze --sarif-output-version Current --recurse --output D:\a\1\s\.gdn\.r\binskim\001\binskim.sarif @D:\a\1\s\.gdn\.r\binskim\001\.gdntoolinput 2023-10-13T11:14:02.9230397Z Analyzing... 2023-10-13T11:14:03.1804055Z 2023-10-13T11:14:03.1804831Z Done. 3 files scanned. 2023-10-13T11:14:03.1810309Z Analysis completed successfully. 2023-10-13T11:14:03.1842896Z 2023-10-13T11:14:03.1868718Z One or more targets was skipped entirely as it was determined to be an invalid target for analysis. Pass --verbose on the command-line for more information. 2023-10-13T11:14:03.1887489Z 2023-10-13T11:14:03.3089507Z Tool run time: 0.6040107 seconds 2023-10-13T11:14:03.3112550Z ------------------------------------------------------------------------------ 2023-10-13T11:14:03.3115247Z BinSkim completed with exit code 0 2023-10-13T11:14:03.3116408Z ------------------------------------------------------------------------------ 2023-10-13T11:14:03.3116944Z 2023-10-13T11:14:03.3117314Z Running AntiMalware 1.7.0 2023-10-13T11:14:05.7809730Z AntiMalware completed with exit code 0 2023-10-13T11:14:05.7810403Z ------------------------------------------------------------------------------ 2023-10-13T11:14:05.7811000Z 2023-10-13T11:14:05.7833644Z Running Bandit 1.6.3.1 2023-10-13T11:14:06.6593431Z Bandit completed with exit code 0 2023-10-13T11:14:06.6594336Z ------------------------------------------------------------------------------ 2023-10-13T11:14:06.6594928Z 2023-10-13T11:14:06.6604038Z Process: 2023-10-13T11:14:06.6611731Z Convert: 2023-10-13T11:14:06.6655902Z Converting any raw tool logs to Sarif format ... 2023-10-13T11:14:06.6745007Z Found 1 logs for tool binskim. 2023-10-13T11:14:06.7341084Z Completed converting raw tool logs to Sarif format. 2023-10-13T11:14:06.7341872Z Import: 2023-10-13T11:14:06.7804244Z ##[error]JsonReaderException: Unexpected character encountered while parsing value: S. Path '', line 0, position 0. 2023-10-13T11:14:06.9828573Z ##[error]MSDO CLI exited with an error exit code: 1 2023-10-13T11:14:06.9936608Z ##[section]Finishing: Microsoft Security for DevOps ```

chrisnielsen-MS commented 8 months ago

Thank you for reporting this @teemukj -- are you still seeing the issue in the latest release (1.10.0) as well?

teemukj commented 6 months ago

Seems to be OK with newer versions.

jaspervdstraten commented 3 months ago

Hi @chrisnielsen-MS, sorry for hijacking this issue. But I am running into the same issue as described above with version 1.11.1

Analyze: Running Terrascan 1.18.0.1

Tool run time: 1.4032052 seconds
    ------------------------------------------------------------------------------
    Terrascan completed with exit code 5
    ------------------------------------------------------------------------------

  Process:
    Convert:
      Converting any raw tool logs to Sarif format ...
      Completed converting raw tool logs to Sarif format.
    Import:
##[error]JsonReaderException: Unexpected character encountered while parsing value: S. Path '', line 0, position 0.
##[error]MSDO CLI exited with an error exit code: 1
Finishing: Microsoft Security DevOps

1andonlyAllen commented 3 months ago

Tool run time: 1.4032052 seconds

Terrascan completed with exit code 5
------------------------------------------------------------------------------

Process: Convert: Converting any raw tool logs to Sarif format ... Completed converting raw tool logs to Sarif format. Import:

[error]JsonReaderException: Unexpected character encountered while parsing value: S. Path '', line 0, position 0.

[error]MSDO CLI exited with an error exit code: 1

Finishing: Microsoft Security DevOps

microsoft / security-devops-azdevops