BUG: Pinned-Dependencies assumes that Dockerfile commands can be parsed as `sh`

[Porges]

Describe the bug Pinned-Dependencies parses commands in Dockerfiles using the sh (https://github.com/mvdan/sh) parser. However, Dockerfiles can use any shell they want using the SHELL statement.

In particular, this means that Dockerfiles designed for Windows targets (and Powershell) will often fail to parse.

Reproduction steps Steps to reproduce the behavior:

1. Create a Dockerfile that uses a non-default SHELL which doesn't parse as sh (an example here)
2. Pinned-Dependencies might fail to parse it Pinned-Dependencies: internal error: error parsing shell code: src/runtime-tools/win64/Dockerfile:1:18: & can only immediately follow a statement

Expected behavior Pinned-Dependencies should not assume that all Dockerfiles contain sh-compatible commands, especially if a SHELL statement is present.

[ckreibich]

Just a heads-up that we've hit this in the Zeek project as well, in this Windows Dockerfile. In our case the complaint is:

Error: check runtime error: Pinned-Dependencies: internal error: error parsing shell code: ci/windows/Dockerfile:1:124: (( can only be used to open an arithmetic cmd

[naveensrinivasan]

https://github.com/ossf/scorecard/issues/2932#issue-1691623575

[danmoseley]

Also for us -- blocks running this on the dotnet/runtime repo.

https://github.com/dotnet/runtime/blob/79c021d65c280020246d1035b0e87ae36f2d36a9/eng/docker/libraries-sdk.windows.Dockerfile#L15

RUN & .\dotnet-install.ps1 -Channel $env:_DOTNET_INSTALL_CHANNEL -Quality daily -InstallDir 'C:/Program Files/dotnet'

[danmoseley]

@afmarcum could you help set expectations on this? Perhaps it's to change "Pinned-Dependencies" to skip lines it can't parse.

Would love to be able to get a scorecard for dotnet/runtime, it's one of the most active repos on Github by their measures.

[pnacht]

@danmoseley I believe this was fixed in #3515, which hasn't entered a release yet.

Running Scorecard with that PR, we get the following output for Pinned-Dependencies for dotnet/runtime:

{
  "details": [
    "Info: Possibly incomplete results: error parsing shell code: & can only immediately follow a statement: eng/docker/libraries-sdk.windows.Dockerfile:14",
    "Info: Possibly incomplete results: error parsing shell code: & can only immediately follow a statement: eng/docker/libraries-sdk.windows.Dockerfile:23",
    "Info: Possibly incomplete results: error parsing shell code: \"fi\" can only be used to end an if: eng/testing/RunnerTemplate.sh:0",
    # ... unpinned things ...
    "Warn: containerImage not pinned by hash: eng/docker/libraries-sdk.windows.Dockerfile:4",
    # ... more unpinned things ...
    "Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned",
    "Info:   0 out of   1 third-party GitHubAction dependencies pinned",
    "Info:   0 out of  18 containerImage dependencies pinned",
    "Info:   0 out of   2 downloadThenRun dependencies pinned",
    "Info:   0 out of   2 npmCommand dependencies pinned"
  ],
  "score": 0,
  "reason": "dependency not pinned by hash detected -- score normalized to 0",
  "name": "Pinned-Dependencies",
  "documentation": {
    "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies",
    "short": "Determines if the project has declared and pinned the dependencies of its build process."
  }
}

Note that the parsing errors on the Dockerfile do not stop Scorecard from detecting that it is unpinned.