Add support for SARIF - Githubissues

mathroule commented 1 year ago

How to output as SARIF using error message formatter?

Is Go template flexible enough to do it or it's easier to output as JSON and then convert it to SARIF?

Thanks

rhysd commented 1 year ago

I don't know the SARIF format. So I can't answer it. For the syntax of Go template, please refer the following document. Basic control flows (e.g. loops, branches, blocks, ...) are supported.

mathroule commented 1 year ago

SARIF (Static Analysis Results Interchange Format) is an OASIS standard supported by GitHub Code Scanning.

rhysd commented 1 year ago

I don't know the details of the specification so I can't answer this question.

mathroule commented 1 year ago

Ok. So I guess there is no plan yet to support SARIF in actionlint?

rhysd commented 1 year ago

Is this a feature request? Since the description of this issue are questions, I thought it just asked questions.

So I guess there is no plan yet to support SARIF in actionlint?

As of now, I have no plan since it may be possible by -format option. If the option can cover SARIF, adding SARIF support separately is unnecessary. Once I understand the option cannot cover SARIF, I would consider how to support it.

mathroule commented 1 year ago

Basically, a SARIF is a JSON file with a structure that can look like this for actionlint:

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": "v1.6.25",
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {
                            "id": "unexpected-keys",
                            "name": "UnexpectedKeys",
                            "defaultConfiguration": {
                                "level": "error"
                            },
                            "properties": {
                                "description": "Unexpected keys",
                                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md#check-unexpected-keys"
                            },
                            "shortDescription": {
                                "text": "Unexpected keys"
                            },
                            "fullDescription": {
                                "text": "Workflow syntax defines what keys can be defined in which mapping object. When other keys are defined, they are simply ignored and don't affect workflow behavior. It means typo in keys is not detected by GitHub."
                            },
                            "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md#check-unexpected-keys"
                        }
                    ]
                }
            },
            "results": [
                {
                    "ruleId": "unexpected-keys",
                    "ruleIndex": 0,
                    "message": {
                        "text": "unexpected key \"cancel-in\" for \"concurrency\" section. expected one of \"cancel-in-progress\", \"group\""
                    },
                    "locations": [
                        {
                            "physicalLocation": {
                                "artifactLocation": {
                                    "uri": ".github/workflows/check-pull-request.yml",
                                    "uriBaseId": "%SRCROOT%"
                                },
                                "region": {
                                    "startLine": 10,
                                    "startColumn": 3,
                                    "endColumn": 12
                                }
                            }
                        }
                    ]
                }
            ]
        }
    ]
}

The SARIF file contains:

Metadata about the linter
The matching rules (can be used by several results)
The results

So I'm not sure if the Go template output would be enough to generate a JSON structure like this.

rhysd commented 1 year ago

Thanks for the explanation. I'm also looking at examples in the spec.

https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html#_Toc34317948

I'm not fully understanding, but

actionlint doesn't have IDs for each checks in the document with the useful descriptions. It is requested by #75, but it is a bit hard since I don't have enough time to work on it. Instead, kind (which is category of errors) may be available.
Go template does not provide a function to encode as URI. Though it is easy to add a function using Template.Funcs

rhysd commented 1 year ago

@mathroule BTW, what is your use case with this feature?

mathroule commented 1 year ago

@mathroule BTW, what is your use case with this feature?

Linting GitHub Actions workflows and uploading the results to GitHub Code Scanning.

rhysd commented 1 year ago

It sounds great. It might be off-topic, but how do you plan to add the support for GitHub Code Scanning once actionlint's -format can output errors in SARIF?

mathroule commented 1 year ago

It sounds great. It might be off-topic, but how do you plan to add the support for GitHub Code Scanning once actionlint's -format can output errors in SARIF?

By using a GitHub Actions: https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github#example-workflow-for-sarif-files-generated-outside-of-a-repository

It assumes the workflow running the lint process, does not contains syntax errors, and can be run on GitHub Actions. Anyway, it's still useful for other workflows.

rhysd commented 1 year ago

I tried some templates for -format action, however it is difficult to enumerate all rules since currently there is no list of all rules in implementation of actionlint. I'll try to add some implementation to extend the error formatter used in -format so that errors can be output in SARIF format.

mathroule commented 1 year ago

Indeed the list of rules must be outputted to be able to generate a SARIF. That's would be great to have it implemented. Many thanks!

rhysd commented 1 year ago

@mathroule

I'm working on this in issue-311 branch.

I tried to output errors in SARIF format as follows. I verified the output with sarif-tools and it was correct. Does the output look good to you?

There are some limitations:

queryURI is fixed because sections in docs.md are not corresponding to each rules
ID and short description are same because actionlint's rules don't have short descriptions

Content of `template.txt`

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": "v1.6.25",
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {{$first := true}}
                        {{range $ = allKinds }}
                            {{if $first}}{{$first = false}}{{else}},{{end}}
                            {
                                "id": {{json $.Name}},
                                "name": {{json $.Name}},
                                "defaultConfiguration": {
                                    "level": "error"
                                },
                                "properties": {
                                    "description": {{json $.Description}},
                                    "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                                },
                                "shortDescription": {
                                    "text": {{json $.Name}}
                                },
                                "fullDescription": {
                                    "text": {{json $.Description}}
                                },
                                "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                            }
                        {{end}}
                    ]
                }
            },
            "results": [
                {{$first := true}}
                {{range $ = .}}
                    {{if $first}}{{$first = false}}{{else}},{{end}}
                    {
                        "ruleId": {{json $.Kind}},
                        "ruleIndex": {{kindIndex $.Kind}},
                        "message": {
                            "text": {{json $.Message}}
                        },
                        "locations": [
                            {
                                "physicalLocation": {
                                    "artifactLocation": {
                                        "uri": {{json $.Filepath}},
                                        "uriBaseId": "%SRCROOT%"
                                    },
                                    "region": {
                                        "startLine": {{$.Line}},
                                        "startColumn": {{$.Column}},
                                        "endColumn": {{$.EndColumn}}
                                    }
                                }
                            }
                        ]
                    }
                {{end}}
            ]
        }
    ]
}

Command

actionlint -format "$(cat template.txt)" testdata/examples/main.yaml

Output

{
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "version": "2.1.0",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "GitHub Actions lint",
          "version": "v1.6.25",
          "informationUri": "https://github.com/rhysd/actionlint",
          "rules": [
            {
              "id": "syntax-check",
              "name": "syntax-check",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub Actions workflow syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "syntax-check"
              },
              "fullDescription": {
                "text": "Checks for GitHub Actions workflow syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "matrix",
              "name": "matrix",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for matrix combinations in \"matrix:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "matrix"
              },
              "fullDescription": {
                "text": "Checks for matrix combinations in \"matrix:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "credentials",
              "name": "credentials",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for credentials in \"services:\" configuration",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "credentials"
              },
              "fullDescription": {
                "text": "Checks for credentials in \"services:\" configuration"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shell-name",
              "name": "shell-name",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell names used for scripts in \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shell-name"
              },
              "fullDescription": {
                "text": "Checks for shell names used for scripts in \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "runner-label",
              "name": "runner-label",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "runner-label"
              },
              "fullDescription": {
                "text": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "events",
              "name": "events",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for workflow trigger events at \"on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "events"
              },
              "fullDescription": {
                "text": "Checks for workflow trigger events at \"on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "job-needs",
              "name": "job-needs",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "job-needs"
              },
              "fullDescription": {
                "text": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "action",
              "name": "action",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "action"
              },
              "fullDescription": {
                "text": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "env-var",
              "name": "env-var",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for environment variables configuration at \"env:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "env-var"
              },
              "fullDescription": {
                "text": "Checks for environment variables configuration at \"env:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "id",
              "name": "id",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for duplication and naming convention of job/step IDs",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "id"
              },
              "fullDescription": {
                "text": "Checks for duplication and naming convention of job/step IDs"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "glob",
              "name": "glob",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for glob syntax used in branch names, tags, and paths",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "glob"
              },
              "fullDescription": {
                "text": "Checks for glob syntax used in branch names, tags, and paths"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "permissions",
              "name": "permissions",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "permissions"
              },
              "fullDescription": {
                "text": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "workflow-call",
              "name": "workflow-call",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "workflow-call"
              },
              "fullDescription": {
                "text": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "expression",
              "name": "expression",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Syntax and semantics checks for expressions embedded with ${{ }} syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "expression"
              },
              "fullDescription": {
                "text": "Syntax and semantics checks for expressions embedded with ${{ }} syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "deprecated-commands",
              "name": "deprecated-commands",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "deprecated-commands"
              },
              "fullDescription": {
                "text": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shellcheck",
              "name": "shellcheck",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell script sources in \"run:\" using shellcheck",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shellcheck"
              },
              "fullDescription": {
                "text": "Checks for shell script sources in \"run:\" using shellcheck"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "pyflakes",
              "name": "pyflakes",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for Python script when \"shell: python\" is configured using Pyflakes",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "pyflakes"
              },
              "fullDescription": {
                "text": "Checks for Python script when \"shell: python\" is configured using Pyflakes"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            }
          ]
        }
      },
      "results": [
        {
          "ruleId": "syntax-check",
          "ruleIndex": 0,
          "message": {
            "text": "unexpected key \"branch\" for \"push\" section. expected one of \"branches\", \"branches-ignore\", \"paths\", \"paths-ignore\", \"tags\", \"tags-ignore\", \"types\", \"workflows\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 5,
                  "endColumn": 11
                }
              }
            }
          ]
        },
        {
          "ruleId": "glob",
          "ruleIndex": 10,
          "message": {
            "text": "character '\\' is invalid for branch and tag names. only special characters [, ?, +, *, \\ ! can be escaped with \\. see `man git-check-ref-format` for more details. note that regular expression is unavailable. note: filter pattern syntax is explained at https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 5,
                  "startColumn": 11,
                  "endColumn": 14
                }
              }
            }
          ]
        },
        {
          "ruleId": "runner-label",
          "ruleIndex": 4,
          "message": {
            "text": "label \"linux-latest\" is unknown. available labels are \"windows-latest\", \"windows-2022\", \"windows-2019\", \"windows-2016\", \"ubuntu-latest\", \"ubuntu-22.04\", \"ubuntu-20.04\", \"ubuntu-18.04\", \"macos-latest\", \"macos-latest-xl\", \"macos-13-xl\", \"macos-13\", \"macos-13.0\", \"macos-12-xl\", \"macos-12\", \"macos-12.0\", \"macos-11\", \"macos-11.0\", \"macos-10.15\", \"self-hosted\", \"x64\", \"arm\", \"arm64\", \"linux\", \"macos\", \"windows\". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 10,
                  "startColumn": 28,
                  "endColumn": 40
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "ruleIndex": 13,
          "message": {
            "text": "\"github.event.head_commit.message\" is potentially untrusted. avoid using it directly in inline scripts. instead, pass it through an environment variable. see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions for more details"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 13,
                  "startColumn": 41,
                  "endColumn": 72
                }
              }
            }
          ]
        },
        {
          "ruleId": "action",
          "ruleIndex": 7,
          "message": {
            "text": "input \"node_version\" is not defined in action \"actions/setup-node@v3\". available inputs are \"always-auth\", \"architecture\", \"cache\", \"cache-dependency-path\", \"check-latest\", \"node-version\", \"node-version-file\", \"registry-url\", \"scope\", \"token\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 17,
                  "startColumn": 11,
                  "endColumn": 23
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "ruleIndex": 13,
          "message": {
            "text": "property \"platform\" is not defined in object type {os: string}"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 21,
                  "startColumn": 20,
                  "endColumn": 34
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "ruleIndex": 13,
          "message": {
            "text": "receiver of object dereference \"permissions\" must be type of object but got \"string\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 22,
                  "startColumn": 17,
                  "endColumn": 51
                }
              }
            }
          ]
        }
      ]
    }
  ]
}

rhysd commented 1 year ago

Oh, I've just understood that ruleIndex is not mandatory.

https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html#_Toc34317644

If ruleIndex can be omitted, the implementation can get much simpler. I'll update the branch.

mathroule commented 1 year ago

Thanks @rhysd that's awesome!

I used this online validator: https://sarifweb.azurewebsites.net/Validation with "GitHub ingestion rules" enabled, and it reported some warnings with the sample output you provided:

Version should follow semantic versioning and become: "version": "1.6.25"
Rules ID and name must be different
Rule ID must be a "stable, opaque identifier" (the SARIF specification (3.49.3) explains the reasons for this)
Rule name is a PascalCase identifier that is understandable to an end user (3.49.7)

About ruleIndex, indeed it can be omitted.

Do you think the template can be directly integrated into actionlint to avoid the need for duplication? And call actionlint with something like that: actionlint -format sarif

rhysd commented 1 year ago

Thank you for the feedback.

Rule ID must be a "stable, opaque identifier" (the SARIF specification (3.49.3) explains the reasons for this)

This is not possible because actionlint doesn't have a stable and opaque identifier for each rule. Each rule has its name only.

Rule name is a PascalCase identifier that is understandable to an end user (3.49.7)

This is possible by adding a new template function. I'll add it.

Do you think the template can be directly integrated into actionlint to avoid the need for duplication?

It's possible, but I don't want to add support for dedicated format because it increases maintenance cost. For example, when version of SARIF is bumped, we need to follow it. So actionlint's policy is to support several formats (e.g. JSON, jSONL, Markdown, SARIF, ...) through -format option and Go template.

rhysd commented 1 year ago

I updated the branch.

Remove rule index as it is optional
Add toPascalCase template function so that rule names are in pascal case

Now it works as follows:

Template file

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": "1.6.25",
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {{$first := true}}
                        {{range $ = allKinds }}
                            {{if $first}}{{$first = false}}{{else}},{{end}}
                            {
                                "id": {{json $.Name}},
                                "name": {{$.Name | toPascalCase | json}},
                                "defaultConfiguration": {
                                    "level": "error"
                                },
                                "properties": {
                                    "description": {{json $.Description}},
                                    "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                                },
                                "shortDescription": {
                                    "text": {{json $.Name}}
                                },
                                "fullDescription": {
                                    "text": {{json $.Description}}
                                },
                                "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                            }
                        {{end}}
                    ]
                }
            },
            "results": [
                {{$first := true}}
                {{range $ = .}}
                    {{if $first}}{{$first = false}}{{else}},{{end}}
                    {
                        "ruleId": {{json $.Kind}},
                        "message": {
                            "text": {{json $.Message}}
                        },
                        "locations": [
                            {
                                "physicalLocation": {
                                    "artifactLocation": {
                                        "uri": {{json $.Filepath}},
                                        "uriBaseId": "%SRCROOT%"
                                    },
                                    "region": {
                                        "startLine": {{$.Line}},
                                        "startColumn": {{$.Column}},
                                        "endColumn": {{$.EndColumn}}
                                    }
                                }
                            }
                        ]
                    }
                {{end}}
            ]
        }
    ]
}

Command

actionlint -format "$(cat template.txt)" testdata/examples/main.yaml

Output (formatted)

{
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "version": "2.1.0",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "GitHub Actions lint",
          "version": "1.6.25",
          "informationUri": "https://github.com/rhysd/actionlint",
          "rules": [
            {
              "id": "action",
              "name": "Action",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "action"
              },
              "fullDescription": {
                "text": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "expression",
              "name": "Expression",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Syntax and semantics checks for expressions embedded with ${{ }} syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "expression"
              },
              "fullDescription": {
                "text": "Syntax and semantics checks for expressions embedded with ${{ }} syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shellcheck",
              "name": "Shellcheck",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell script sources in \"run:\" using shellcheck",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shellcheck"
              },
              "fullDescription": {
                "text": "Checks for shell script sources in \"run:\" using shellcheck"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "events",
              "name": "Events",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for workflow trigger events at \"on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "events"
              },
              "fullDescription": {
                "text": "Checks for workflow trigger events at \"on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shell-name",
              "name": "ShellName",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell names used for scripts in \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shell-name"
              },
              "fullDescription": {
                "text": "Checks for shell names used for scripts in \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "job-needs",
              "name": "JobNeeds",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "job-needs"
              },
              "fullDescription": {
                "text": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "permissions",
              "name": "Permissions",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "permissions"
              },
              "fullDescription": {
                "text": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "workflow-call",
              "name": "WorkflowCall",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "workflow-call"
              },
              "fullDescription": {
                "text": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "syntax-check",
              "name": "SyntaxCheck",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub Actions workflow syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "syntax-check"
              },
              "fullDescription": {
                "text": "Checks for GitHub Actions workflow syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "runner-label",
              "name": "RunnerLabel",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "runner-label"
              },
              "fullDescription": {
                "text": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "env-var",
              "name": "EnvVar",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for environment variables configuration at \"env:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "env-var"
              },
              "fullDescription": {
                "text": "Checks for environment variables configuration at \"env:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "id",
              "name": "Id",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for duplication and naming convention of job/step IDs",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "id"
              },
              "fullDescription": {
                "text": "Checks for duplication and naming convention of job/step IDs"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "glob",
              "name": "Glob",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for glob syntax used in branch names, tags, and paths",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "glob"
              },
              "fullDescription": {
                "text": "Checks for glob syntax used in branch names, tags, and paths"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "pyflakes",
              "name": "Pyflakes",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for Python script when \"shell: python\" is configured using Pyflakes",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "pyflakes"
              },
              "fullDescription": {
                "text": "Checks for Python script when \"shell: python\" is configured using Pyflakes"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "credentials",
              "name": "Credentials",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for credentials in \"services:\" configuration",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "credentials"
              },
              "fullDescription": {
                "text": "Checks for credentials in \"services:\" configuration"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "deprecated-commands",
              "name": "DeprecatedCommands",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "deprecated-commands"
              },
              "fullDescription": {
                "text": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "matrix",
              "name": "Matrix",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for matrix combinations in \"matrix:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "matrix"
              },
              "fullDescription": {
                "text": "Checks for matrix combinations in \"matrix:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            }
          ]
        }
      },
      "results": [
        {
          "ruleId": "syntax-check",
          "message": {
            "text": "unexpected key \"branch\" for \"push\" section. expected one of \"branches\", \"branches-ignore\", \"paths\", \"paths-ignore\", \"tags\", \"tags-ignore\", \"types\", \"workflows\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 5,
                  "endColumn": 11
                }
              }
            }
          ]
        },
        {
          "ruleId": "glob",
          "message": {
            "text": "character '\\' is invalid for branch and tag names. only special characters [, ?, +, *, \\ ! can be escaped with \\. see `man git-check-ref-format` for more details. note that regular expression is unavailable. note: filter pattern syntax is explained at https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 5,
                  "startColumn": 11,
                  "endColumn": 14
                }
              }
            }
          ]
        },
        {
          "ruleId": "runner-label",
          "message": {
            "text": "label \"linux-latest\" is unknown. available labels are \"windows-latest\", \"windows-2022\", \"windows-2019\", \"windows-2016\", \"ubuntu-latest\", \"ubuntu-22.04\", \"ubuntu-20.04\", \"ubuntu-18.04\", \"macos-latest\", \"macos-latest-xl\", \"macos-13-xl\", \"macos-13\", \"macos-13.0\", \"macos-12-xl\", \"macos-12\", \"macos-12.0\", \"macos-11\", \"macos-11.0\", \"macos-10.15\", \"self-hosted\", \"x64\", \"arm\", \"arm64\", \"linux\", \"macos\", \"windows\". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 10,
                  "startColumn": 28,
                  "endColumn": 40
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "message": {
            "text": "\"github.event.head_commit.message\" is potentially untrusted. avoid using it directly in inline scripts. instead, pass it through an environment variable. see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions for more details"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 13,
                  "startColumn": 41,
                  "endColumn": 72
                }
              }
            }
          ]
        },
        {
          "ruleId": "action",
          "message": {
            "text": "input \"node_version\" is not defined in action \"actions/setup-node@v3\". available inputs are \"always-auth\", \"architecture\", \"cache\", \"cache-dependency-path\", \"check-latest\", \"node-version\", \"node-version-file\", \"registry-url\", \"scope\", \"token\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 17,
                  "startColumn": 11,
                  "endColumn": 23
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "message": {
            "text": "property \"platform\" is not defined in object type {os: string}"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 21,
                  "startColumn": 20,
                  "endColumn": 34
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "message": {
            "text": "receiver of object dereference \"permissions\" must be type of object but got \"string\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 22,
                  "startColumn": 17,
                  "endColumn": 51
                }
              }
            }
          ]
        }
      ]
    }
  ]
}

mathroule commented 1 year ago

Thanks @rhysd!

One last thing: it would be great to dynamically output the GitHub Actions lint version. For instance, having something like this: "version": "1.6.25", =>"version": {{ getVersion }}.

Also, the code snippet can be outputted like this in the region block:

"region": {
    "startLine": {{$.Line}},
    "startColumn": {{$.Column}},
    "endColumn": {{$.EndColumn}},
    "snippet": {
        "text": {{json $.Snippet}}
    }
}

rhysd commented 1 year ago

Thanks. That's good point. I'll implement it tomorrow.

rhysd commented 1 year ago

I added getVersion function at 0f67af81b140bb8f42d35398f4e0b99e070bee5b. Now

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": {{ getVersion | json }},
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {{$first := true}}
                        {{range $ = allKinds }}
                            {{if $first}}{{$first = false}}{{else}},{{end}}
                            {
                                "id": {{json $.Name}},
                                "name": {{$.Name | toPascalCase | json}},
                                "defaultConfiguration": {
                                    "level": "error"
                                },
                                "properties": {
                                    "description": {{json $.Description}},
                                    "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                                },
                                "fullDescription": {
                                    "text": {{json $.Description}}
                                },
                                "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                            }
                        {{end}}
                    ]
                }
            },
            "results": [
                {{$first := true}}
                {{range $ = .}}
                    {{if $first}}{{$first = false}}{{else}},{{end}}
                    {
                        "ruleId": {{json $.Kind}},
                        "message": {
                            "text": {{json $.Message}}
                        },
                        "locations": [
                            {
                                "physicalLocation": {
                                    "artifactLocation": {
                                        "uri": {{json $.Filepath}},
                                        "uriBaseId": "%SRCROOT%"
                                    },
                                    "region": {
                                        "startLine": {{$.Line}},
                                        "startColumn": {{$.Column}},
                                        "endColumn": {{$.EndColumn}},
                                        "snippet": {{json $.Snippet}}
                                    }
                                }
                            }
                        ]
                    }
                {{end}}
            ]
        }
    ]
}

formats errors as follows:

{
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "version": "2.1.0",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "GitHub Actions lint",
          "version": "1.6.25",
          "informationUri": "https://github.com/rhysd/actionlint",
          "rules": [
            {
              "id": "workflow-call",
              "name": "WorkflowCall",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "fullDescription": {
                "text": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            // ...
          ]
        }
      },
      "results": [
        {
          "ruleId": "syntax-check",
          "message": {
            "text": "unexpected key \"branch\" for \"push\" section. expected one of \"branches\", \"branches-ignore\", \"paths\", \"paths-ignore\", \"tags\", \"tags-ignore\", \"types\", \"workflows\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 5,
                  "endColumn": 11,
                  "snippet": "    branch: main\n    ^~~~~~~"
                }
              }
            }
          ]
        },
        // ...
      ]
    }
  ]
}

mathroule commented 1 year ago

Thanks @rhysd for taking the time to work on SARIF support. I really appreciate your help and dedication!

Any plan to merge issue-311 branch and release a new version of actionlint?

rhysd commented 1 year ago

@mathroule Nothing blocks merging the branch. I just confirmed the output looks good to you before doing it. I'll merge it soon.

I need some other issues to address before releasing next version. So it would take a while for shipping the next release.

rhysd commented 1 year ago

I merged the branch. The SARIF template is put here: https://github.com/rhysd/actionlint/blob/main/testdata/format/sarif_template.txt

mathroule commented 1 year ago

@rhysd the template https://github.com/rhysd/actionlint/blob/main/testdata/format/sarif_template.txt contains an error with snippet, it should be:

                                        "snippet": {
                                            "text": {{json $.Snippet}}
                                        }

Instead of:

                                        "snippet": {{json $.Snippet}}

rhysd commented 1 year ago

Ah, thanks for catching it.

rhysd / actionlint

Add support for SARIF #311

Content of `template.txt`

Command

Output

Template file

Command

Output (formatted)

rhysd / actionlint

Add support for SARIF #311

Content of template.txt

Command

Output

Template file

Command

Output (formatted)

Content of `template.txt`