VEX documents not taken into account when `--fail-on` is set

[ferozsalam]

What happened:

Using the --fail-on flag doesn't appear to take into account the contents of a VEX document that has been provided using the --vex flag - even if several CVEs have been marked as not_affected in the VEX document, the command still fails when the --fail-on flag is set.

This is inconsistent with the behaviour shown when a .grype.yaml file is used with an ignore block defined, despite VEX documents and the ignore block having similar purposes.

What you expected to happen:

If Grype is invoked with both --vex and --fail-on set, and all CVEs in the scanned image are defined as not_affected in the VEX document, I would expect the command to pass successfully, as it does when all such CVEs are defined in a .grype.yaml ignore block.

How to reproduce it (as minimally and precisely as possible):

Create the following VEX doc and save it as test-vex.json:

{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://openvex.dev/docs/public/vex-cb58dca06abcf3da095d3f5e6126128aeb39b145e57d4eac468483eb75b5c014",
  "author": "Unknown Author",
  "timestamp": "2023-12-18T12:00:54.292255Z",
  "last_updated": "2023-12-18T12:01:58.35081Z",
  "version": 4,
  "statements": [
    {
      "vulnerability": {
        "name": "CVE-2023-42366"
      },
      "timestamp": "2023-12-18T12:00:54.292256Z",
      "products": [
        {
          "@id": "pkg:oci/busybox"
        }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path"
    },
    {
      "vulnerability": {
        "name": "CVE-2023-42365"
      },
      "timestamp": "2023-12-18T12:01:17.799359Z",
      "products": [
        {
          "@id": "pkg:oci/busybox"
        }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path"
    },
    {
      "vulnerability": {
        "name": "CVE-2023-42364"
      },
      "timestamp": "2023-12-18T12:01:41.069995Z",
      "products": [
        {
          "@id": "pkg:oci/busybox"
        }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path"
    },
    {
      "vulnerability": {
        "name": "CVE-2023-42363"
      },
      "timestamp": "2023-12-18T12:01:58.35081Z",
      "products": [
        {
          "@id": "pkg:oci/busybox"
        }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path"
    }
  ]
}

Using the --vex flag with the JSON file above to filter out the CVEs works as expected:

➜  ~ grype busybox:1.36.1@sha256:b920f79ed6b477f64b7cbce0c4458ba09ffdbf8c8dbd025bc8bf5baa999255e3 --vex test-vex.json
 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                     busybox:1.36.1@sha256:b920f79ed6b477f64b7cbce0c4458ba09ffdbf8c8dbd025bc8bf5baa999255e3
 ✔ Parsed image                                                                    sha256:a6969cfac53d7d994eabab99ef8080b8495ae7ddd34731128718316797c2244b
 ✔ Cataloged packages              [1 packages]
 ✔ Scanned for vulnerabilities     [0 vulnerability matches]
   ├── by severity: 0 critical, 0 high, 4 medium, 0 low, 0 negligible
   └── by status:   0 fixed, 4 not-fixed, 4 ignored
No vulnerabilities found

However, if you set the --fail-on flag, the --vex flag is ignored and the vulnerabilities reappear.

➜  ~ grype busybox:1.36.1@sha256:b920f79ed6b477f64b7cbce0c4458ba09ffdbf8c8dbd025bc8bf5baa999255e3 --vex test-vex.json --fail-on medium
 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                     busybox:1.36.1@sha256:b920f79ed6b477f64b7cbce0c4458ba09ffdbf8c8dbd025bc8bf5baa999255e3
 ✔ Parsed image                                                                    sha256:a6969cfac53d7d994eabab99ef8080b8495ae7ddd34731128718316797c2244b
 ✔ Cataloged packages              [1 packages]
 ✘ Scan for vulnerabilities        [4 vulnerability matches]
   ├── by severity: 0 critical, 0 high, 4 medium, 0 low, 0 negligible
   └── by status:   0 fixed, 4 not-fixed, 0 ignored
NAME     INSTALLED  FIXED-IN  TYPE    VULNERABILITY   SEVERITY
busybox  1.36.1               binary  CVE-2023-42366  Medium
busybox  1.36.1               binary  CVE-2023-42365  Medium
busybox  1.36.1               binary  CVE-2023-42364  Medium
busybox  1.36.1               binary  CVE-2023-42363  Medium
1 error occurred:
        * discovered vulnerabilities at or above the severity threshold

However, if a .grype.yaml is created with the same CVEs listed in an ignore block, those CVEs are ignored. Create a .grype.yaml with the following:

ignore:
- vulnerability: CVE-2023-42366
- vulnerability: CVE-2023-42365
- vulnerability: CVE-2023-42364
- vulnerability: CVE-2023-42363

And then invoke Grype with --fail-on set:

➜  ~ grype busybox:1.36.1@sha256:b920f79ed6b477f64b7cbce0c4458ba09ffdbf8c8dbd025bc8bf5baa999255e3 --fail-on medium
 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                     busybox:1.36.1@sha256:b920f79ed6b477f64b7cbce0c4458ba09ffdbf8c8dbd025bc8bf5baa999255e3
 ✔ Parsed image                                                                    sha256:a6969cfac53d7d994eabab99ef8080b8495ae7ddd34731128718316797c2244b
 ✔ Cataloged packages              [1 packages]
 ✔ Scanned for vulnerabilities     [0 vulnerability matches]
   ├── by severity: 0 critical, 0 high, 4 medium, 0 low, 0 negligible
   └── by status:   0 fixed, 4 not-fixed, 0 ignored
No vulnerabilities found

cc @puerco

Environment:

• Output of grype version:

➜  ~ grype version
Application:         grype
Version:             0.73.4
BuildDate:           2023-11-30T14:32:13Z
GitCommit:           a4bced160291e81dbdeaa415d20e64b8741ee00d
GitDescription:      v0.73.4
Platform:            darwin/arm64
GoVersion:           go1.21.4
Compiler:            gc
Syft Version:        v0.98.0
Supported DB Schema: 5

• OS (e.g: cat /etc/os-release or similar): MacOS Ventura 13.6.3

[tgerla]

Hi @ferozsalam, thanks for the report! We will take a look.

[ferozsalam]

Hey @tgerla if the team agrees that this is a bug and that the correct behaviour should be as described in 'What you expected to happen', then I'm happy to take a look at fixing this myself.

[tgerla]

Hi @ferozsalam, that would be excellent. Yes, we agree that this is a bug and your described behavior is probably the right approach. If you need any guidance or anything, just let us know here or in our community Slack and we can help you out. Thanks much!