Open sj-williams opened 10 months ago
We also encountered this and I believe it is caused because (at least in the rootfs mode) the trivy binary is copied into the image being scanned. Trivy then scans itself and detects its own vulnerabilities, these vulnerabilities are then reported against the image being scanned.
Helm release: 0.16.4
@sj-williams in your case its also rootfs
mode ?
@chen-keinan yes I can confirm I am setting this value to rootfs
in my chart config. Following @andrew-gropyus 's comment I've also verified that the trivy
binary shipping with my chart version 0.13
confirms these 4 CVEs on a local trivy scan of the image: ghcr.io/aquasecurity/trivy:0.39.0
Comparing this to a local scan of the trivy
image provided with latest Helm release 0.18.4
(ghcr.io/aquasecurity/trivy:0.45.1
) I can see that of those 4 HIGH CVEs mentioned above, only google.golang.org/grpc
remains present.
I have a task to test upgrading operator to latest Helm release and confirm this is indeed the cause. Will report back.
@chen-keinan have carried out testing of upgrading operator to latest Helm release 0.18.4
, and can confirm I now see 3 less HIGH level CVEs reported per vulnerability report across cluster.
The remaining common HIGH CVE being caught is google.golang.org/grpc
(https://github.com/advisories/GHSA-m425-mq94-257g) which also lines up with a local trivy scan of ghcr.io/aquasecurity/trivy:0.45.1
, which would support @andrew-gropyus findings.
Is this an issue we are stuck with using rootfs
config?
@sj-williams thanks for the feedback , could be related to usage of rootfs
in operator , I will review it and update
@andrew-gropyus @sj-williams do you mind sharing the command line which you are using on trivy cli
and also an image (public one) where this issue reproduce
@chen-keinan I can get you this sometime later this week.
@chen-keinan actually I'm not sure I fully understood what you were asking of myself in your previous comment.
In any case, here are some steps which can reproduce this issue. Here we are pretending to be a trivy operator scan-job running in rootfs mode.
# First choose a sample image that contains no known vulnerabilities
trivy image nginx:stable-alpine3.17-slim
# To use rootfs we have to extract the image's filesystem
mkdir tmp
cd tmp
docker run -d --name sample nginx:stable-alpine3.17-slim
docker export sample | tar -x
docker stop sample
docker rm sample
# Verify that there are still no vulnerabilities when using the rootfs mode
trivy rootfs .
# Emulate trivy-operator's scan-job init container's behaviour of copying trivy into the image being scanned
# Use a version of trivy that has known vulnerabilities
mkdir var/trivyoperator
curl -L https://github.com/aquasecurity/trivy/releases/download/v0.45.1/trivy_0.45.1_macOS-64bit.tar.gz | tar -xz > var/trivyoperator/trivy
# Run trivy in rootfs mode again and see the vulnerabilities
trivy rootfs .
@chen-keinan actually I'm not sure I fully understood what you were asking of myself in your previous comment.
In any case, here are some steps which can reproduce this issue. Here we are pretending to be a trivy operator scan-job running in rootfs mode.
# First choose a sample image that contains no known vulnerabilities trivy image nginx:stable-alpine3.17-slim # To use rootfs we have to extract the image's filesystem mkdir tmp cd tmp docker run -d --name sample nginx:stable-alpine3.17-slim docker export sample | tar -x docker stop sample docker rm sample # Verify that there are still no vulnerabilities when using the rootfs mode trivy rootfs . # Emulate trivy-operator's scan-job init container's behaviour of copying trivy into the image being scanned # Use a version of trivy that has known vulnerabilities mkdir var/trivyoperator curl -L https://github.com/aquasecurity/trivy/releases/download/v0.45.1/trivy_0.45.1_macOS-64bit.tar.gz | tar -xz > var/trivyoperator/trivy # Run trivy in rootfs mode again and see the vulnerabilities trivy rootfs .
@andrew-gropyus thanks , this what I was looking for steps for reproduce
@andrew-gropyus I have made a simple test by deploying this pod to k8s:
---
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app.kubernetes.io/name: nginx
app: nginx
spec:
containers:
- image: nginx:stable-alpine3.17-slim
name: app
imagePullPolicy: Always
and install operator via :
helm install trivy-operator aqua/trivy-operator \
--namespace trivy-system \
--create-namespace \
--set="trivy.command=rootfs" \
--set="trivyOperator.scanJobPodTemplateContainerSecurityContext.runAsUser=0" \
--version 0.19.1
check for vuln report
kubectl get vulns --all-namespaces -o wide
after scan I got this results :
NAMESPACE NAME REPOSITORY TAG SCANNER AGE CRITICAL HIGH MEDIUM LOW UNKNOWN
default pod-nginx-app library/nginx stable-alpine3.17-slim Trivy 27s 0 0 0 0 0
kube-system daemonset-kindnet-kindnet-cni kindest/kindnetd v20210326-1e038dc5 Trivy 38s 19 55 48 81 2
its same results as trivy rootfs
cli
Am I missing anything ?
@chen-keinan I suppose because the 0.19.1 version of the helm chart uses trivy 0.47.0, which has no vulnerabilities. Can you either override the trivy version to be 0.45.1 (maybe --set "trivy.image.tag=0.45.1"
) or use an older version of the helm chart?
@chen-keinan , I believe @andrew-gropyus 's observation regarding the vulnerability free Trivy image is correct here. My workaround as of Helm release 0.18.4
has been to override the trivy binary image release to 0.47.0
- following a verification via local scan that this image is vulnerability free. I've got this flagged as a temporary fix for now as I expect as future vulnerabilities are detected in trivy binary we will see this propagate through to cluster scans again.
This issue is stale because it has been labeled with inactivity.
What steps did you take and what happened:
I am seeing a large number of trivy-operator generated vulnerability reports flagging CVEs for the following resources:
These are flagging on scans of many different images across multiple namespaces in my cluster, seemingly irrespective of whether the scanned image has a dependency on the flagged resources.
Local scanning of said images does not produce the same CVE finding results as operator in-cluster scans.
What did you expect to happen:
trivy-operator in-cluster vulnerabilty report image scan CVE findings to match those of a local trivy cli scan.
Anything else you would like to add:
Environment:
0.13
EKS v1.24