Summary counters are not accurate

dwertent commented 7 months ago

What happened: I sanned the following:

% grype quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6 --by-cve
 ✔ Scanned for vulnerabilities     [30 vulnerability matches]
   ├── by severity: 1 critical, 18 high, 11 medium, 0 low, 0 negligible

Here is the list of vulnerabilities:

stdlib             go1.21.0             go-module  CVE-2023-39320  Critical
golang.org/x/net   v0.13.0    0.17.0    go-module  CVE-2023-44487  High
golang.org/x/net   v0.13.0    0.17.0    go-module  CVE-2023-39325  High
golang.org/x/net   v0.14.0    0.17.0    go-module  CVE-2023-44487  High
golang.org/x/net   v0.14.0    0.17.0    go-module  CVE-2023-39325  High
k8s.io/kubernetes  v1.28.0    1.28.4    go-module  CVE-2023-5528   High
k8s.io/kubernetes  v1.28.0    1.28.1    go-module  CVE-2023-3955   High
k8s.io/kubernetes  v1.28.0    1.28.1    go-module  CVE-2023-3676   High
stdlib             go1.20.7             go-module  CVE-2023-45285  High
stdlib             go1.20.7             go-module  CVE-2023-45283  High
stdlib             go1.20.7             go-module  CVE-2023-44487  High
stdlib             go1.20.7             go-module  CVE-2023-39325  High
stdlib             go1.20.7             go-module  CVE-2023-39323  High
stdlib             go1.21.0             go-module  CVE-2023-45285  High
stdlib             go1.21.0             go-module  CVE-2023-45283  High
stdlib             go1.21.0             go-module  CVE-2023-44487  High
stdlib             go1.21.0             go-module  CVE-2023-39325  High
stdlib             go1.21.0             go-module  CVE-2023-39323  High
stdlib             go1.21.0             go-module  CVE-2023-39322  High
stdlib             go1.21.0             go-module  CVE-2023-39321  High
stdlib             go1.21.0             go-module  CVE-2023-45284  Medium
stdlib             go1.21.0             go-module  CVE-2023-39326  Medium
stdlib             go1.21.0             go-module  CVE-2023-39319  Medium
stdlib             go1.21.0             go-module  CVE-2023-39318  Medium
stdlib             go1.20.7             go-module  CVE-2023-45284  Medium
stdlib             go1.20.7             go-module  CVE-2023-39326  Medium
stdlib             go1.20.7             go-module  CVE-2023-39319  Medium
stdlib             go1.20.7             go-module  CVE-2023-39318  Medium

As you can tell, there are 19 High and 8 Medium, not 18 high, and 11 medium as it says in the summary.

What you expected to happen: The numbers should add up.

How to reproduce it (as minimally and precisely as possible): Run the following:

grype quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6 --by-cve

Anything else we need to know?:

Full output:

% grype quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6 --by-cve
 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                                                                                                                                                                                                                                        quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6
 ✔ Parsed image                                                                                                                                                                                                                                                                                                   sha256:b747f07011a2c82631683d826b2275a6fcfbe6dfb5c26ac48d69446767cd7511
 ✔ Cataloged packages              [176 packages]
 ✔ Scanned for vulnerabilities     [30 vulnerability matches]
   ├── by severity: 1 critical, 18 high, 11 medium, 0 low, 0 negligible
   └── by status:   9 fixed, 21 not-fixed, 0 ignored
[0003]  WARN some package(s) are missing CPEs. This may result in missing vulnerabilities. You may autogenerate these using: --add-cpes-if-none
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/api, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/api" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/apimachinery, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/apimachinery" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/cli-runtime, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/cli-runtime" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/client-go, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/client-go" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/component-base, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/component-base" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/component-helpers, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/component-helpers" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/kubectl, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/kubectl" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/metrics, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/metrics" ver="(devel)": Malformed version: (devel)
[0003]  WARN could not match by package language (package=Pkg(type=go-module, name=github.com/tigera/cc-operator, version=(devel), upstreams=0)): matcher failed to parse version pkg="github.com/tigera/cc-operator" ver="(devel)": Malformed version: (devel)
NAME               INSTALLED  FIXED-IN  TYPE       VULNERABILITY   SEVERITY
golang.org/x/net   v0.13.0    0.17.0    go-module  CVE-2023-44487  High
golang.org/x/net   v0.13.0    0.17.0    go-module  CVE-2023-39325  High
golang.org/x/net   v0.14.0    0.17.0    go-module  CVE-2023-44487  High
golang.org/x/net   v0.14.0    0.17.0    go-module  CVE-2023-39325  High
k8s.io/kubernetes  v1.28.0    1.28.4    go-module  CVE-2023-5528   High
k8s.io/kubernetes  v1.28.0    1.28.1    go-module  CVE-2023-3955   High
k8s.io/kubernetes  v1.28.0    1.28.1    go-module  CVE-2023-3676   High
stdlib             go1.20.7             go-module  CVE-2023-45285  High
stdlib             go1.20.7             go-module  CVE-2023-45283  High
stdlib             go1.20.7             go-module  CVE-2023-44487  High
stdlib             go1.20.7             go-module  CVE-2023-39325  High
stdlib             go1.20.7             go-module  CVE-2023-39323  High
stdlib             go1.20.7             go-module  CVE-2023-45284  Medium
stdlib             go1.20.7             go-module  CVE-2023-39326  Medium
stdlib             go1.20.7             go-module  CVE-2023-39319  Medium
stdlib             go1.20.7             go-module  CVE-2023-39318  Medium
stdlib             go1.21.0             go-module  CVE-2023-39320  Critical
stdlib             go1.21.0             go-module  CVE-2023-45285  High
stdlib             go1.21.0             go-module  CVE-2023-45283  High
stdlib             go1.21.0             go-module  CVE-2023-44487  High
stdlib             go1.21.0             go-module  CVE-2023-39325  High
stdlib             go1.21.0             go-module  CVE-2023-39323  High
stdlib             go1.21.0             go-module  CVE-2023-39322  High
stdlib             go1.21.0             go-module  CVE-2023-39321  High
stdlib             go1.21.0             go-module  CVE-2023-45284  Medium
stdlib             go1.21.0             go-module  CVE-2023-39326  Medium
stdlib             go1.21.0             go-module  CVE-2023-39319  Medium
stdlib             go1.21.0             go-module  CVE-2023-39318  Medium
A newer version of grype is available for download: 0.74.2 (installed version is 0.73.4)

And here are the results without the --by-cve flag:

 % grype quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6
 ⠴ Vulnerability DB                ━━━━━━━━━━━━━━━━━━━━  [checking for update]
 ⠴ Vulnerability DB                ━━━━━━━━━━━━━━━━━━━━  [checking for update]
 ✔ Vulnerability DB                [no update available]
 ✔ Vulnerability DB                [no update available]
 ✔ Vulnerability DB                [no update available]
 ✔ Vulnerability DB                [no update available]
 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                                                                                                                           quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6
 ✔ Parsed image                                                                                                                                                                                      sha256:b747f07011a2c82631683d826b2275a6fcfbe6dfb5c26ac48d69446767cd7511
 ✔ Cataloged packages              [176 packages]
 ✔ Scanned for vulnerabilities     [30 vulnerability matches]
   ├── by severity: 1 critical, 18 high, 11 medium, 0 low, 0 negligible
   └── by status:   9 fixed, 21 not-fixed, 0 ignored
[0002]  WARN some package(s) are missing CPEs. This may result in missing vulnerabilities. You may autogenerate these using: --add-cpes-if-none
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/api, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/api" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/apimachinery, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/apimachinery" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/cli-runtime, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/cli-runtime" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/client-go, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/client-go" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/component-base, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/component-base" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/component-helpers, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/component-helpers" ver="(devel)": Malformed version: (dev
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/kubectl, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/kubectl" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=./staging/src/k8s.io/metrics, version=(devel), upstreams=0)): matcher failed to parse version pkg="./staging/src/k8s.io/metrics" ver="(devel)": Malformed version: (devel)
[0002]  WARN could not match by package language (package=Pkg(type=go-module, name=github.com/tigera/cc-operator, version=(devel), upstreams=0)): matcher failed to parse version pkg="github.com/tigera/cc-operator" ver="(devel)": Malformed version: (devel)
NAME               INSTALLED  FIXED-IN  TYPE       VULNERABILITY        SEVERITY
golang.org/x/net   v0.13.0    0.17.0    go-module  GHSA-4374-p667-p6c8  High
golang.org/x/net   v0.13.0    0.17.0    go-module  GHSA-qppj-fm5r-hxr3  Medium
golang.org/x/net   v0.14.0    0.17.0    go-module  GHSA-4374-p667-p6c8  High
golang.org/x/net   v0.14.0    0.17.0    go-module  GHSA-qppj-fm5r-hxr3  Medium
k8s.io/kubernetes  v1.28.0    1.28.1    go-module  GHSA-q78c-gwqw-jcmc  High
k8s.io/kubernetes  v1.28.0    1.28.4    go-module  GHSA-hq6q-c2x6-hmch  High
k8s.io/kubernetes  v1.28.0    1.28.1    go-module  GHSA-7fxm-f474-hf8w  High
stdlib             go1.20.7             go-module  CVE-2023-45285       High
stdlib             go1.20.7             go-module  CVE-2023-45283       High
stdlib             go1.20.7             go-module  CVE-2023-44487       High
stdlib             go1.20.7             go-module  CVE-2023-39325       High
stdlib             go1.20.7             go-module  CVE-2023-39323       High
stdlib             go1.20.7             go-module  CVE-2023-45284       Medium
stdlib             go1.20.7             go-module  CVE-2023-39326       Medium
stdlib             go1.20.7             go-module  CVE-2023-39319       Medium
stdlib             go1.20.7             go-module  CVE-2023-39318       Medium
stdlib             go1.21.0             go-module  CVE-2023-39320       Critical
stdlib             go1.21.0             go-module  CVE-2023-45285       High
stdlib             go1.21.0             go-module  CVE-2023-45283       High
stdlib             go1.21.0             go-module  CVE-2023-44487       High
stdlib             go1.21.0             go-module  CVE-2023-39325       High
stdlib             go1.21.0             go-module  CVE-2023-39323       High
stdlib             go1.21.0             go-module  CVE-2023-39322       High
stdlib             go1.21.0             go-module  CVE-2023-39321       High
stdlib             go1.21.0             go-module  CVE-2023-45284       Medium
stdlib             go1.21.0             go-module  CVE-2023-39326       Medium
stdlib             go1.21.0             go-module  CVE-2023-39319       Medium
stdlib             go1.21.0             go-module  CVE-2023-39318       Medium
A newer version of grype is available for download: 0.74.2 (installed version is 0.73.4)

As you can tell, also here the summary does not add up.

Environment:

Output of grype version:

% grype version
Application:         grype
Version:             0.73.4
BuildDate:           2023-11-30T14:22:34Z
GitCommit:           brew
GitDescription:      [not provided]
Platform:            darwin/arm64
GoVersion:           go1.21.4
Compiler:            gc
Syft Version:        v0.98.0
Supported DB Schema: 5

OS (e.g: cat /etc/os-release or similar): macos M1

tgerla commented 7 months ago

Hi @dwertent, thank you for the detailed report. We do have some circumstances where the summary count won't line up with the default table view output, specifically when there are duplicate rows in the table referring to the same vulnerability against the same package but installed in different places on your system. The JSON output should be correct relative to the the summary counts, but we agree that this is confusing.

Here is a similar issue with our proposed fix: annotate de-duplicated rows in the table output with something like (2) to indicate that a vulnerability has a duplicate: https://github.com/anchore/grype/issues/1327

We need to do some more investigation to make sure that our explanation is correct, we'll add this issue to our backlog to look at when we are able. Thanks again!

wagoodman commented 7 months ago

I quick check looking at the JSON output too shows that there is a count issue still:

$ grype quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6 --by-cve -o json | jq '.matches[] | select(.vulnerability.severity == "Medium") | .vulnerability.id ' | wc -l

 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                                                             quay.io/tigera/cc-operator@sha256:d6451279b74c73a669d2901c321d386163f09fdc3cccabaf4d34b2ad065ea6c6
 ✔ Parsed image                                                                                                                        sha256:b747f07011a2c82631683d826b2275a6fcfbe6dfb5c26ac48d69446767cd7511
 ✔ Scanned for vulnerabilities     [30 vulnerability matches]
   ├── by severity: 1 critical, 18 high, 11 medium, 0 low, 0 negligible
   └── by status:   9 fixed, 21 not-fixed, 0 ignored
...
       8

wagoodman commented 6 months ago

For this image at this time, here are all of the matches with a severity of medium (13)

GHSA-qppj-fm5r-hxr3     golang.org/x/net@v0.13.0
GHSA-qppj-fm5r-hxr3     golang.org/x/net@v0.14.0
GHSA-qppj-fm5r-hxr3     golang.org/x/net@v0.14.0
GHSA-8r3f-844c-mc37     google.golang.org/protobuf@v1.30.0
GHSA-8r3f-844c-mc37     google.golang.org/protobuf@v1.31.0
CVE-2023-39318          stdlib@go1.20.7
CVE-2023-39319          stdlib@go1.20.7
CVE-2023-39326          stdlib@go1.20.7
CVE-2023-45284          stdlib@go1.20.7
CVE-2023-39318          stdlib@go1.21.0
CVE-2023-39319          stdlib@go1.21.0
CVE-2023-39326          stdlib@go1.21.0
CVE-2023-45284          stdlib@go1.21.0

Taking a look at the non-stdlib matches, when not using --by-cve you'll see:

NAME                        INSTALLED  FIXED-IN  TYPE       VULNERABILITY        SEVERITY
golang.org/x/net            v0.13.0    0.17.0    go-module  GHSA-4374-p667-p6c8  High
golang.org/x/net            v0.13.0    0.17.0    go-module  GHSA-qppj-fm5r-hxr3  Medium
golang.org/x/net            v0.14.0    0.17.0    go-module  GHSA-4374-p667-p6c8  High
golang.org/x/net            v0.14.0    0.17.0    go-module  GHSA-qppj-fm5r-hxr3  Medium
google.golang.org/protobuf  v1.30.0    1.33.0    go-module  GHSA-8r3f-844c-mc37  Medium
google.golang.org/protobuf  v1.31.0    1.33.0    go-module  GHSA-8r3f-844c-mc37  Medium
k8s.io/kubernetes           v1.28.0    1.28.1    go-module  GHSA-q78c-gwqw-jcmc  High
k8s.io/kubernetes           v1.28.0    1.28.4    go-module  GHSA-hq6q-c2x6-hmch  High
k8s.io/kubernetes           v1.28.0    1.28.1    go-module  GHSA-7fxm-f474-hf8w  High

But with --by-cve you'll see:

golang.org/x/net            v0.13.0    0.17.0    go-module  CVE-2023-44487  High
golang.org/x/net            v0.13.0    0.17.0    go-module  CVE-2023-39325  High
golang.org/x/net            v0.14.0    0.17.0    go-module  CVE-2023-44487  High
golang.org/x/net            v0.14.0    0.17.0    go-module  CVE-2023-39325  High
google.golang.org/protobuf  v1.30.0    1.33.0    go-module  CVE-2024-24786  Unknown
google.golang.org/protobuf  v1.31.0    1.33.0    go-module  CVE-2024-24786  Unknown
k8s.io/kubernetes           v1.28.0    1.28.4    go-module  CVE-2023-5528   High
k8s.io/kubernetes           v1.28.0    1.28.1    go-module  CVE-2023-3955   High
k8s.io/kubernetes           v1.28.0    1.28.1    go-module  CVE-2023-3676   High

Note that the severities are different for the CVEs than the GHSA's for the same matches.

This is mostly due to the way we are implementing --by-cve, which I think will need to be heavily changed. Today this is a presentation concern... we make no changes to the matching process but instead after the matching is done transform non-CVEs to CVEs (if possible) and coalesce the set of transformed matches. I think this is where most of the issue is occuring.

However, there is still one more issue: we search for matches without looking for duplicate packages first:

[0000] TRACE searching for vulnerability matches package=pkg:golang/golang.org/x/net@v0.14.0
[0000] DEBUG found 2 vulnerabilities package=pkg:golang/golang.org/x/net@v0.14.0
[0000] DEBUG   ├── namespace=github:language:go vuln=GHSA-4374-p667-p6c8
[0000] DEBUG   └── namespace=github:language:go vuln=GHSA-qppj-fm5r-hxr3
[0000] TRACE searching for vulnerability matches package=pkg:golang/golang.org/x/net@v0.14.0
[0000] DEBUG found 2 vulnerabilities package=pkg:golang/golang.org/x/net@v0.14.0
[0000] DEBUG   ├── namespace=github:language:go vuln=GHSA-4374-p667-p6c8
[0000] DEBUG   └── namespace=github:language:go vuln=GHSA-qppj-fm5r-hxr3

This is a debug log snippet showing the back-to-back (duplicate) lookup. This hints at either:

more intelligent tracking to look for already discovered results
or more intelligent deduplication logic before attempting searches

anchore / grype

Summary counters are not accurate #1673