Not seeing any vulnerabilities (or errors) when scanning SBOMs with known vulnerabilities

[luhring]

I'm seeing some cases where bomber doesn't appear to be performing a vulnerability scan on the supplied SBOM, but it still reports "no vulnerabilities found" and exits 0. I'd recommend paying careful attention when reporting "no vulnerabilities found" to users, to avoid creating a false sense of security and eroding trust in the tool.

I installed bomber using the Homebrew instructions on the README. And here I'm using Syft version 0.54.0.

$ syft -q ubuntu:latest -o 'spdx-json=./ubuntu.spdx.json'
$ env | grep 'BOMBER'
BOMBER_PROVIDER_USERNAME=<redacted>
BOMBER_PROVIDER_TOKEN=<redacted>
$ bomber scan ./ubuntu.spdx.json

 ██▄ ▄▀▄ █▄ ▄█ ██▄ ██▀ █▀▄
 █▄█ ▀▄▀ █ ▀ █ █▄█ █▄▄ █▀▄

DKFM - DevOps Kung Fu Mafia
https://github.com/devops-kung-fu/bomber
Version: 0.0.1

Uses vulnerability information provided by the Sonatype OSS Index
https://ossindex.sonatype.org

■ Scanning 0 packages for vulnerabilities...

No vulnerabilities found!

■ Done
$ echo $?
0

In contrast, when I use Grype, it scans the SBOM for vulnerabilities correctly:

$ grype -q ./ubuntu.spdx.json
NAME          INSTALLED                 FIXED-IN          TYPE  VULNERABILITY   SEVERITY
coreutils     8.32-4.1ubuntu1                             deb   CVE-2016-2781   Low
libc-bin      2.35-0ubuntu3.1                             deb   CVE-2016-20013  Negligible
libc6         2.35-0ubuntu3.1                             deb   CVE-2016-20013  Negligible
libgmp10      2:6.2.1+dfsg-3ubuntu1                       deb   CVE-2021-43618  Low
libgnutls30   3.7.3-4ubuntu1            3.7.3-4ubuntu1.1  deb   CVE-2022-2509   Medium
libncurses6   6.3-2                                       deb   CVE-2022-29458  Negligible
libncursesw6  6.3-2                                       deb   CVE-2022-29458  Negligible
libpcre2-8-0  10.39-3build1                               deb   CVE-2022-1586   Low
libpcre2-8-0  10.39-3build1                               deb   CVE-2022-1587   Low
libpcre3      2:8.39-13ubuntu0.22.04.1                    deb   CVE-2017-11164  Negligible
libtinfo6     6.3-2                                       deb   CVE-2022-29458  Negligible
login         1:4.8.1-2ubuntu2                            deb   CVE-2013-4235   Low
ncurses-base  6.3-2                                       deb   CVE-2022-29458  Negligible
ncurses-bin   6.3-2                                       deb   CVE-2022-29458  Negligible
passwd        1:4.8.1-2ubuntu2                            deb   CVE-2013-4235   Low
perl-base     5.34.0-3ubuntu1                             deb   CVE-2020-16156  Medium
tar           1.34+dfsg-1build3                           deb   CVE-2019-9923   Low
zlib1g        1:1.2.11.dfsg-2ubuntu9                      deb   CVE-2022-37434  Medium

[luhring]

Full SBOM attached: sbom.tar.gz

[djschleen]

@luhring Looks like a few things happening here. The first one I'm going to look at is why bomber doesn't think this is a valid SBoM, I'm going to add some more debug logging and trace through the load of this file. It looks like it didn't get to the PURL parsing routines. It should at the very least say how many packages it found.

The second thing is that bomber passes out the PURLS to the providers, in the case right now its OSS Index. OSS Index doesn't support the deb ecosystem so once bomber would get the PURLs from the SBOM, nothing would be found. That said, I'm going to be implementing OSV this week as the default provider, with OSS Index being an alternate. That's going to bring in deb scanning, and a bunch of other ecosystems.

I'll use the provided SBOM to test all this with

[luhring]

Thanks, sounds great! 🙏

[djschleen]

@luhring So I found two issues, the first is that bomber didn't think the file was a valid SPDX file. I corrected this in my dj-wip branch, and will release that. The big issue is the SPDX format in general. When the ubuntu-latest.spdx.json file was generated, it didn't put any purls in the output. The way SPDX does purls is through the External References (which is really strange). I generated the same command in cyclonedx format and got the purls in my latest code, but nothing was found (I used the OSV provider that I'm writing up now.

Still looking into this, but may just focus on code rather than OS package scanning for bomber if I go down a rabbit hole. I am implementing snyk in the newest code as well, so I'll find out if that provider tosses any info back.