Integrating Rancher VEX files into Aqua VEX Hub

[knqyf263]

Hello Rancher team,

We're reaching out from the Trivy team to discuss the possibility of integrating some of your VEX files into Aqua VEX Hub. We greatly appreciate your adoption of the VEX Repository and are pleased to see it being used as intended.

Background

The Aqua VEX Hub operates on a trust model where VEX files issued by software maintainers are recognized and trusted. We encourage third-party vendors to publish their own VEX repositories, as Rancher does. Trivy provides flexibility by allowing users to set priorities for VEX repositories, enabling them to trust different VEX issuers as needed.

Upon reviewing the Rancher VEX Hub, we noticed that several VEX files appear to be issued in your capacity as a maintainer. Specifically, we identified VEX files for

• pkg:golang/github.com/rancher/rancher
• pkg:golang/github.com/k3s-io/k3s
• etc.

Proposed Collaboration

We would like to propose registering these maintainer-issued VEX files in the Aqua VEX Hub. This integration would benefit users by transparently reducing noise in Trivy scans, as the Aqua VEX Hub is enabled by default in Trivy.

To facilitate this integration, we suggest two possible methods:

1. Store VEX files under the .vex/ directory in each source repository and add the PURLs to the VEX Hub Crawler.
   • For example: github.com/rancher/rancher/.vex/scan.openvex.json.
   • You can find more details in our documentation: https://github.com/aquasecurity/vexhub-crawler?tab=readme-ov-file#registering-purls
2. Register the PURLs and Rancher VEX Hub URL directly with the VEX Hub Crawler.
   • While not explicitly documented, we also support specifying the VEX location directly.

If you're interested in pursuing this collaboration, we're more than willing to assist with adding these VEX files to the Aqua VEX Hub.

[macedogm]

Hey @knqyf263. Thanks for your message and for noticing our efforts around VEX. We truly appreciate Trivy's work on this and the amazing idea of VEX Hub, which we are using extensively to remove known false-positives CVEs in our codes and images. We would love to collaborate on this and share our reports to Trivy's VEX Hub, so then more users can benefit from them.

We have a central automation in place, with automatic and manual VEX, from where we ran our scans and generate the reports. Given this current model, the best solution to share the reports seems to be:

2. Register the PURLs and Rancher VEX Hub URL directly with the VEX Hub Crawler.
   • While not explicitly documented, we also support specifying the VEX location directly.

In the future we might add the reports to each repo under the .vex/ directory, but this is not something planned right now.

All the VEX reports under:

• Harvester: https://github.com/rancher/vexhub/tree/main/pkg/golang/github.com/harvester
• K3s: https://github.com/rancher/vexhub/tree/main/pkg/golang/github.com/k3s-io
• Longhorn: https://github.com/rancher/vexhub/tree/main/pkg/golang/github.com/longhorn
• Rancher: https://github.com/rancher/vexhub/tree/main/pkg/golang/github.com/rancher

Are fully owned and vetted by SUSE Rancher, so we can safely share them with Trivy's VEX Hub.

We can implement a daily automation to submit PRs every time that they are updated or, as you mentioned, register them in https://github.com/aquasecurity/vexhub-crawler/blob/main/crawler.yaml, which seems to be the ideal method, right? If yes, can you share examples on how to register the direct VEX Hub reports URLs, please?

[knqyf263]

We can implement a daily automation to submit PRs every time that they are updated or, as you mentioned, register them in https://github.com/aquasecurity/vexhub-crawler/blob/main/crawler.yaml, which seems to be the ideal method, right?

Yes. Adding the following lines to the manifest should work.

  golang:
    - namespace: github.com/rancher
      name: rancher
      url: https://github.com/rancher/vexhub/tree/main/pkg/golang/github.com/rancher

But I found a small problem. I'll update you once we fix it. https://github.com/aquasecurity/vexhub-crawler/issues/27

[macedogm]

@knqyf263 thanks! Will wait for your ping.

[macedogm]

Upstream PRs to integrate with Trivy's VEX Hub:

• [x] Rancher - https://github.com/aquasecurity/vexhub-crawler/pull/29
• [ ] K3s - https://github.com/aquasecurity/vexhub-crawler/pull/30
• [ ] Harvester - https://github.com/aquasecurity/vexhub-crawler/pull/31
• [ ] Longhorn - https://github.com/aquasecurity/vexhub-crawler/pull/32

[macedogm]

@knqyf263 do you want me to submit the PRs to add our other projects ^ ?

[knqyf263]

Yes, it would be appreciated!

[knqyf263]

I found one problem. The subcomponent in Rancher VEX files has v prefixes. https://github.com/aquasecurity/vexhub/blob/25a6a0337648fe8f82daaff57544edfa2949277c/pkg/golang/github.com/rancher/rancher/scan.openvex.json#L696

However, Trivy doesn't have the v prefix in PURL.

$ wget -q https://raw.githubusercontent.com/rancher/rancher/46cc864e3b17aabb9bf87d0cdd30592cd6f608c9/go.mod
$ trivy fs --format cyclonedx ./go.mod | grep 'pkg:golang/github.com/docker/docker@'
2024-10-24T15:34:38+04:00       INFO    "--format cyclonedx" disables security scanning. Specify "--scanners vuln" explicitly if you want to include vulnerabilities in the "cyclonedx" report.
2024-10-24T15:34:40+04:00       INFO    Number of language-specific files       num=1
      "bom-ref": "pkg:golang/github.com/docker/docker@20.10.27%2Bincompatible",
      "purl": "pkg:golang/github.com/docker/docker@20.10.27%2Bincompatible",
      "ref": "pkg:golang/github.com/docker/docker@20.10.27%2Bincompatible",
        "pkg:golang/github.com/docker/docker@20.10.27%2Bincompatible",

Due to this discrepancy, vulnerabilities are not correctly suppressed. The spec uses commit hashes and doesn't mention versions. The Go v prefix is always controversial. (ref. https://github.com/aquasecurity/trivy/discussions/7242)

It looks like there is a discussion about the leading v prefix, but at the moment, v doesn't seem to be supported, as you mentioned. Are you planning to delete the prefixes from Rancher VEX? Or do you want Trivy to handle both cases?

[knqyf263]

Other tools may not handle both cases, so adding both to VEX may be one of the options.

          "subcomponents": [
            {
              "@id": "pkg:golang/github.com/docker/docker@20.10.27+incompatible",
              "@id": "pkg:golang/github.com/docker/docker@v20.10.27+incompatible"
            }

[macedogm]

Thanks for bringing this topic and it's a funny/sad discussion about Go's v prefix, as it generates these kind of issues.

When we first noticed this, we started to use our own forked version of govulncheck while this situation isn't defined by upstream. See the diff. Our fork is adding the affected dependency and version. This forked version correctly generates a VEX entry that works with Trivy and is a valid OpenVEX report. We plan to remove our fork once https://github.com/golang/go/issues/68152 is implemented (hopefully).

Trivy and VEX in Rancher are used to scan our released container images. For example, if you pass this VEX file to Trivy and scan the image rancher/rke2-cloud-provider:v1.29.8-build20240910, Trivy will correctly remove the VEXed entries even with the v.

Image scan without VEX

> trivy -q i -s CRITICAL,HIGH rancher/rke2-cloud-provider:v1.29.8-build20240910

usr/local/bin/rke2-cloud-provider (gobinary)

Total: 2 (HIGH: 1, CRITICAL: 1)

┌──────────────────────────────────────────────────────────────┬────────────────┬──────────┬────────┬──────────────────────┬─────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│                           Library                            │ Vulnerability  │ Severity │ Status │  Installed Version   │          Fixed Version          │                            Title                            │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┼────────┼──────────────────────┼─────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ github.com/docker/docker                                     │ CVE-2024-41110 │ CRITICAL │ fixed  │ v25.0.5+incompatible │ 23.0.15, 26.1.5, 27.1.1, 25.0.6 │ moby: Authz zero length regression                          │
│                                                              │                │          │        │                      │                                 │ https://avd.aquasec.com/nvd/cve-2024-41110                  │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┤        ├──────────────────────┼─────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ go.opentelemetry.io/contrib/instrumentation/google.golang.o- │ CVE-2023-47108 │ HIGH     │        │ v0.45.0              │ 0.46.0                          │ opentelemetry-go-contrib: DoS vulnerability in otelgrpc due │
│ rg/grpc/otelgrpc                                             │                │          │        │                      │                                 │ to unbound cardinality metrics                              │
│                                                              │                │          │        │                      │                                 │ https://avd.aquasec.com/nvd/cve-2023-47108                  │
└──────────────────────────────────────────────────────────────┴────────────────┴──────────┴────────┴──────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────────────────────┘

Image scan with VEX

> trivy -q i -s CRITICAL,HIGH --vex scan.openvex.json --show-suppressed rancher/rke2-cloud-provider:v1.29.8-build20240910

usr/local/bin/rke2-cloud-provider (gobinary)

Total: 0 (HIGH: 0, CRITICAL: 0)

Suppressed Vulnerabilities (Total: 2)

┌──────────────────────────────────────────────────────────────┬────────────────┬──────────┬──────────────┬─────────────────────────────────────┬───────────────────┐
│                           Library                            │ Vulnerability  │ Severity │    Status    │              Statement              │      Source       │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┼──────────────┼─────────────────────────────────────┼───────────────────┤
│ github.com/docker/docker                                     │ CVE-2024-41110 │ CRITICAL │ not_affected │ vulnerable_code_not_in_execute_path │ scan.openvex.json │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┤              │                                     │                   │
│ go.opentelemetry.io/contrib/instrumentation/google.golang.o- │ CVE-2023-47108 │ HIGH     │              │                                     │                   │
│ rg/grpc/otelgrpc                                             │                │          │              │                                     │                   │
└──────────────────────────────────────────────────────────────┴────────────────┴──────────┴──────────────┴─────────────────────────────────────┴───────────────────┘

What I just noticed now is that if I do the same scan, but instead of scanning the image I scan the repo, then it will not remove the VEXed entry, because the affected version reported by Trivy doesn't have the v prefix.

Repo scan without VEX

> trivy -q repo -s CRITICAL,HIGH --tag v1.29.8-build20240910 https://github.com/rancher/image-build-rke2-cloud-provider

go.mod (gomod)

Total: 2 (HIGH: 1, CRITICAL: 1)

┌──────────────────────────────────────────────────────────────┬────────────────┬──────────┬────────┬─────────────────────┬─────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│                           Library                            │ Vulnerability  │ Severity │ Status │  Installed Version  │          Fixed Version          │                            Title                            │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┼────────┼─────────────────────┼─────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ github.com/docker/docker                                     │ CVE-2024-41110 │ CRITICAL │ fixed  │ 25.0.4+incompatible │ 23.0.15, 26.1.5, 27.1.1, 25.0.6 │ moby: Authz zero length regression                          │
│                                                              │                │          │        │                     │                                 │ https://avd.aquasec.com/nvd/cve-2024-41110                  │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┤        ├─────────────────────┼─────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ go.opentelemetry.io/contrib/instrumentation/google.golang.o- │ CVE-2023-47108 │ HIGH     │        │ 0.35.0              │ 0.46.0                          │ opentelemetry-go-contrib: DoS vulnerability in otelgrpc due │
│ rg/grpc/otelgrpc                                             │                │          │        │                     │                                 │ to unbound cardinality metrics                              │
│                                                              │                │          │        │                     │                                 │ https://avd.aquasec.com/nvd/cve-2023-47108                  │
└──────────────────────────────────────────────────────────────┴────────────────┴──────────┴────────┴─────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────────────────────┘

Repo scan with VEX

> trivy -q repo -s CRITICAL,HIGH --tag v1.29.8-build20240910 --vex scan.openvex.json --show-suppressed https://github.com/rancher/image-build-rke2-cloud-provider

go.mod (gomod)

Total: 2 (HIGH: 1, CRITICAL: 1)

┌──────────────────────────────────────────────────────────────┬────────────────┬──────────┬────────┬─────────────────────┬─────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│                           Library                            │ Vulnerability  │ Severity │ Status │  Installed Version  │          Fixed Version          │                            Title                            │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┼────────┼─────────────────────┼─────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ github.com/docker/docker                                     │ CVE-2024-41110 │ CRITICAL │ fixed  │ 25.0.4+incompatible │ 23.0.15, 26.1.5, 27.1.1, 25.0.6 │ moby: Authz zero length regression                          │
│                                                              │                │          │        │                     │                                 │ https://avd.aquasec.com/nvd/cve-2024-41110                  │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┤        ├─────────────────────┼─────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ go.opentelemetry.io/contrib/instrumentation/google.golang.o- │ CVE-2023-47108 │ HIGH     │        │ 0.35.0              │ 0.46.0                          │ opentelemetry-go-contrib: DoS vulnerability in otelgrpc due │
│ rg/grpc/otelgrpc                                             │                │          │        │                     │                                 │ to unbound cardinality metrics                              │
│                                                              │                │          │        │                     │                                 │ https://avd.aquasec.com/nvd/cve-2023-47108                  │
└──────────────────────────────────────────────────────────────┴────────────────┴──────────┴────────┴─────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────────────────────┘

Do you also see the difference between the affected version in repo scan versus image scan?

I think that we can add the VEX entry for the versions without the v prefix too. Not sure if you should also handle this in Trivy, but there is some difference between both scan modes.

[macedogm]

Perhaps the best solution for now is for us to add VEX entries for both v prefix and without it, so it will work with Trivy's repo and image scan.

I honestly have no idea what would be the right way, given the upstream situation and the PURL spec as you mentioned.

[knqyf263]

Oh, yes. You're right. We currently remove the v prefix only for go.mod. After fixing this bug, they should be consistent.

I also have no idea about the canonical version format in PURL, but we can use versions with the prefix for now. Once PURL defines the spec, we can conform to that.

[knqyf263]

Rancher VEX documents newly registered in Aqua VEX Hub work like a charm with container images, as you pointed out 🎉

$ trivy image --scanners vuln --vex repo --show-suppressed rancher/rke2-cloud-provider:v1.29.8-build20240910
2024-10-24T17:57:28+04:00       INFO    [vex] No need to check repository updates       repo="default"
2024-10-24T17:57:28+04:00       INFO    [vuln] Vulnerability scanning is enabled
2024-10-24T17:57:55+04:00       INFO    Number of language-specific files       num=1
2024-10-24T17:57:55+04:00       INFO    [gobinary] Detecting vulnerabilities...

usr/local/bin/rke2-cloud-provider (gobinary)
============================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

Suppressed Vulnerabilities (Total: 2)

┌──────────────────────────────────────────────────────────────┬────────────────┬──────────┬──────────────┬─────────────────────────────────────┬──────────────────────────────────────────┐
│                           Library                            │ Vulnerability  │ Severity │    Status    │              Statement              │                  Source                  │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┼──────────────┼─────────────────────────────────────┼──────────────────────────────────────────┤
│ github.com/docker/docker                                     │ CVE-2024-41110 │ CRITICAL │ not_affected │ vulnerable_code_not_in_execute_path │ VEX Repository: default                  │
│                                                              │                │          │              │                                     │ (https://github.com/aquasecurity/vexhub) │
├──────────────────────────────────────────────────────────────┼────────────────┼──────────┤              │                                     │                                          │
│ go.opentelemetry.io/contrib/instrumentation/google.golang.o- │ CVE-2023-47108 │ HIGH     │              │                                     │                                          │
│ rg/grpc/otelgrpc                                             │                │          │              │                                     │                                          │
└──────────────────────────────────────────────────────────────┴────────────────┴──────────┴──────────────┴─────────────────────────────────────┴──────────────────────────────────────────┘

Thanks for your help! Once we fix https://github.com/aquasecurity/trivy/issues/7711, I'm sure it will work with go.mod as well.

[macedogm]

Oh, yes. You're right. We currently remove the v prefix only for go.mod. After fixing this bug, they should be consistent.

Thanks for sharing that issue and PR. I wasn't aware of it. I'll monitor it for when it's merged.

I also have no idea about the canonical version format in PURL, but we can use versions with the prefix for now. Once PURL defines the spec, we can conform to that.

Agree. When the definition is done we will update our reports to match the upstream spec.

[macedogm]

Rancher VEX documents newly registered in Aqua VEX Hub work like a charm with container images, as you pointed out 🎉

Nice!

Thanks for your help! Once we fix https://github.com/aquasecurity/trivy/issues/7711, I'm sure it will work with go.mod as well.

Indeed! And we thank you and Aqua for developing this amazing feature. It helps a lot to fight the false-positive CVE noise and fatigue.

[macedogm]

I'll be submitting the PRs to add the other projects until tomorrow, then I'll close this issue.

[knqyf263]

Thanks for sharing that issue and PR. I wasn't aware of it. I'll monitor it for when it's merged.

Hopefully, we'll include it in v0.57.0.

And we thank you and Aqua for developing this amazing feature. It helps a lot to fight the false-positive CVE noise and fatigue.

We thank you, too. We believed VEX Hub would reduce noise, but it would be useless if no one generated VEX and added it to VEX Hub.

I'll be submitting the PRs to add the other projects until tomorrow, then I'll close this issue.

Great. Thanks for your contribution!

[macedogm]

@knqyf263 PTAL:

• https://github.com/aquasecurity/vexhub-crawler/pull/30
• https://github.com/aquasecurity/vexhub-crawler/pull/31
• https://github.com/aquasecurity/vexhub-crawler/pull/32

[knqyf263]

All merged, and Rancher VEXes are available now on Aqua VEX Hub! Thank you so much!

I'd mention Rancher VEX Hub in my talk next month. Please let me know if you wouldn't like me to do that.