Discrepancy between trivy cli CVE findings and trivy-operator in-cluster vulnerability report findings

[sj-williams]

What steps did you take and what happened:

I am seeing a large number of trivy-operator generated vulnerability reports flagging CVEs for the following resources:

github.com/docker/docker
github.com/docker/distribution
github.com/sigstore/rekor
google.golang.org/grpc

These are flagging on scans of many different images across multiple namespaces in my cluster, seemingly irrespective of whether the scanned image has a dependency on the flagged resources.

Local scanning of said images does not produce the same CVE finding results as operator in-cluster scans.

What did you expect to happen:

trivy-operator in-cluster vulnerabilty report image scan CVE findings to match those of a local trivy cli scan.

Anything else you would like to add:

Environment:

• Trivy-Operator version - Helm release: 0.13
• Kubernetes version : EKS v1.24

[andrew-gropyus]

We also encountered this and I believe it is caused because (at least in the rootfs mode) the trivy binary is copied into the image being scanned. Trivy then scans itself and detects its own vulnerabilities, these vulnerabilities are then reported against the image being scanned.

Helm release: 0.16.4

[chen-keinan]

@sj-williams in your case its also rootfs mode ?

[sj-williams]

@chen-keinan yes I can confirm I am setting this value to rootfs in my chart config. Following @andrew-gropyus 's comment I've also verified that the trivy binary shipping with my chart version 0.13 confirms these 4 CVEs on a local trivy scan of the image: ghcr.io/aquasecurity/trivy:0.39.0

Comparing this to a local scan of the trivy image provided with latest Helm release 0.18.4 (ghcr.io/aquasecurity/trivy:0.45.1) I can see that of those 4 HIGH CVEs mentioned above, only google.golang.org/grpc remains present.

I have a task to test upgrading operator to latest Helm release and confirm this is indeed the cause. Will report back.

[sj-williams]

@chen-keinan have carried out testing of upgrading operator to latest Helm release 0.18.4, and can confirm I now see 3 less HIGH level CVEs reported per vulnerability report across cluster.

The remaining common HIGH CVE being caught is google.golang.org/grpc (https://github.com/advisories/GHSA-m425-mq94-257g) which also lines up with a local trivy scan of ghcr.io/aquasecurity/trivy:0.45.1, which would support @andrew-gropyus findings.

Is this an issue we are stuck with using rootfs config?

[chen-keinan]

@sj-williams thanks for the feedback , could be related to usage of rootfs in operator , I will review it and update

[chen-keinan]

@andrew-gropyus @sj-williams do you mind sharing the command line which you are using on trivy cli and also an image (public one) where this issue reproduce

[andrew-gropyus]

@chen-keinan I can get you this sometime later this week.

[andrew-gropyus]

@chen-keinan actually I'm not sure I fully understood what you were asking of myself in your previous comment.

In any case, here are some steps which can reproduce this issue. Here we are pretending to be a trivy operator scan-job running in rootfs mode.

# First choose a sample image that contains no known vulnerabilities
trivy image nginx:stable-alpine3.17-slim

# To use rootfs we have to extract the image's filesystem
mkdir tmp
cd tmp
docker run -d --name sample nginx:stable-alpine3.17-slim
docker export sample | tar -x
docker stop sample
docker rm sample

# Verify that there are still no vulnerabilities when using the rootfs mode
trivy rootfs .

# Emulate trivy-operator's scan-job init container's behaviour of copying trivy into the image being scanned
# Use a version of trivy that has known vulnerabilities
mkdir var/trivyoperator
curl -L https://github.com/aquasecurity/trivy/releases/download/v0.45.1/trivy_0.45.1_macOS-64bit.tar.gz | tar -xz > var/trivyoperator/trivy

# Run trivy in rootfs mode again and see the vulnerabilities
trivy rootfs .

[chen-keinan]

@chen-keinan actually I'm not sure I fully understood what you were asking of myself in your previous comment.

In any case, here are some steps which can reproduce this issue. Here we are pretending to be a trivy operator scan-job running in rootfs mode.

# First choose a sample image that contains no known vulnerabilities
trivy image nginx:stable-alpine3.17-slim

# To use rootfs we have to extract the image's filesystem
mkdir tmp
cd tmp
docker run -d --name sample nginx:stable-alpine3.17-slim
docker export sample | tar -x
docker stop sample
docker rm sample

# Verify that there are still no vulnerabilities when using the rootfs mode
trivy rootfs .

# Emulate trivy-operator's scan-job init container's behaviour of copying trivy into the image being scanned
# Use a version of trivy that has known vulnerabilities
mkdir var/trivyoperator
curl -L https://github.com/aquasecurity/trivy/releases/download/v0.45.1/trivy_0.45.1_macOS-64bit.tar.gz | tar -xz > var/trivyoperator/trivy

# Run trivy in rootfs mode again and see the vulnerabilities
trivy rootfs .

@andrew-gropyus thanks , this what I was looking for steps for reproduce

[chen-keinan]

@andrew-gropyus I have made a simple test by deploying this pod to k8s:

---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx
    app: nginx
spec:
  containers:
    - image: nginx:stable-alpine3.17-slim
      name: app
      imagePullPolicy: Always

and install operator via :

helm install trivy-operator aqua/trivy-operator \
        --namespace trivy-system \
        --create-namespace \
        --set="trivy.command=rootfs" \
        --set="trivyOperator.scanJobPodTemplateContainerSecurityContext.runAsUser=0" \
        --version 0.19.1

check for vuln report

kubectl get vulns --all-namespaces  -o wide

after scan I got this results :

NAMESPACE            NAME                                                   REPOSITORY                       TAG                      SCANNER   AGE   CRITICAL   HIGH   MEDIUM   LOW   UNKNOWN
default              pod-nginx-app                                    library/nginx                    stable-alpine3.17-slim   Trivy     27s   0          0      0        0     0
kube-system          daemonset-kindnet-kindnet-cni                          kindest/kindnetd                 v20210326-1e038dc5       Trivy     38s   19         55     48       81    2

its same results as trivy rootfs cli

Am I missing anything ?

[andrew-gropyus]

@chen-keinan I suppose because the 0.19.1 version of the helm chart uses trivy 0.47.0, which has no vulnerabilities. Can you either override the trivy version to be 0.45.1 (maybe --set "trivy.image.tag=0.45.1") or use an older version of the helm chart?

[sj-williams]

@chen-keinan , I believe @andrew-gropyus 's observation regarding the vulnerability free Trivy image is correct here. My workaround as of Helm release 0.18.4 has been to override the trivy binary image release to 0.47.0 - following a verification via local scan that this image is vulnerability free. I've got this flagged as a temporary fix for now as I expect as future vulnerabilities are detected in trivy binary we will see this propagate through to cluster scans again.

[github-actions[bot]]

This issue is stale because it has been labeled with inactivity.