Open vpetersson opened 3 months ago
vpetersson
commented
3 months ago @CISA-SBOM-Community/sbom-generation-tiger-team I've posted some benchmarks for container and application generation. Please review and let me know what your thoughts. The qualification methodology is rather primitive but should hopefully help guide us at least.
vpetersson
commented
2 months ago | Tool | Format | Packages | Unique Packages | Duplication % |
|---|---|---|---|---|
| Syft | CycloneDX | 192 | 172 | 10.42% |
| Trivy | CycloneDX | 180 | 178 | 1.11% |
| Syft | SPDX | 192 | 173 | 9.90% |
| Tvivy | SPDX | 181 | 172 | 4.97% |
$ jq '.components[] | .name' syft_container-sbom_cyclonedx.json | wc -l
192
$ jq '.components[] | .name' syft_container-sbom_cyclonedx.json | uniq | wc -l
173$ jq '.packages[] | .name' syft_container-sbom_spdx.json | wc -l
192
$ jq '.packages[] | .name' syft_container-sbom_spdx.json | uniq | wc -l
173$ jq '.components[] | .name' trivy_container-sbom_cyclonedx.json | wc -l
180
$ jq '.components[] | .name' trivy_container-sbom_cyclonedx.json | uniq | wc -l
178$ jq '.packages[] | .name' trivy_container-sbom_spdx.json | wc -l
181
$ jq '.packages[] | .name' trivy_container-sbom_spdx.json | uniq | wc -l
172What we can conclude here is that both tools evaluated generated duplicates.
| Tool | Format | Packages | Unique Packages | Duplication % |
|---|---|---|---|---|
| cyclonedx-python | CycloneDX | 3 | 3 | 0% |
| sbom4python | CycloneDX | 3 | 3 | 0% |
| Syft | CycloneDX | 3 | 3 | 0% |
| Trivy | CycloneDX | 4 | 4 | 0% |
| sbom4python | SPDXD | 3 | 3 | 0% |
| Syft | SPDXD | 4 | 4 | 0% |
| Trivy | SPDXD | 5 | 5 | 0% |
$ for i in *cyclonedx*; do echo $i; jq '.components[] | .name' $i | wc -l; jq '.components[] | .name' $i | uniq | wc -l; done
cyclonedx-python_application-sbom_cyclonedx.json
3
3
sbom4python_application-sbom_cyclonedx.json
3
3
syft_application-sbom_cyclonedx.json
3
3
trivy_application-sbom_cyclonedx.json
4
4$ for i in *spdx*; do echo $i; jq '.packages[] | .name' $i | wc -l; jq '.packages[] | .name' $i | uniq | wc -l; done
sbom4python_application-sbom_spdx.json
3
3
syft_application-sbom_spdx.json
4
4
trivy_application-sbom_spdx.json
5
5Much like the findings in https://github.com/CISA-SBOM-Community/SBOM-Generation/pull/15, Trivy appears to generate less duplicate packages. The findings for the Application SBOM is less interesting.
The objective here isn't necessarily to do a deep dive into the output of these tools. The view is that they should be a good starting point and to be replaceable in the pipeline as they evolve.
With this in mind, I will proceed with trivy.
vpetersson
commented
2 months ago @CISA-SBOM-Community/sbom-generation-tiger-team review. I've implemented the first few steps. Please review the data in the comment above. If anyone has any arguments for why we should not use trivy for both the container and application step, please let me know. If not, I'm going to consider that part Phase 1 resolved.
For the next step, we need to find a tool for creating a hierarchy SBOM that includes both the container and application SBOMs. Note that "merging" these into a single SBOM isn't possible as they are of two different SBOM types (container and application). Thus a tool for creating a hierarchy is needed.
Tools that will be evaluated include bomctl and bomasm. Neither of these tools can do this in the current version but future versions promises this ability. Thus Phase 1 is somewhat blocked by this.
djmoch
commented
2 months ago I'm good with moving forward with Trivy.
Tracker for
Container Image with Python application.Work to be carried out in https://github.com/CISA-SBOM-Community/SBOM-Generation/pull/4
Todo
merginggenerating hierarchical SBOMsmerginghierarchyContainer SBOM Generation
Tools Evaluated
As per the discussion in the Tiger Team, the tools qualify as per the qualification criteria (citation needed).
Before we generate the result, we first need to build the Docker image:
Result
syft phase-1-python -o cyclonedx-json \| jq > syft_container-sbom_cyclonedx.jsonjqfor improved readabilitysyft phase-1-python -o spdx-json \| jq > syft_container-sbom_spdx.jsonjqfor improved readabilitytrivy image --format cyclonedx --output trivy_container-sbom_cyclonedx.json phase-1-pythontrivy image --format spdx-json --output trivy_container-sbom_spdx.json phase-1-pythonWe need to dive deeper dive into the quality of these SBOMs, but based on the amount of data picked up (measured by LOC),
syftappears to pick up a lot more.Some rudamentary automated assesment using sbomdiff (0.5.4) yielded the following:
sbomdiff --sbom spdx syft_container-sbom_spdx.json trivy_container-sbom_spdx.json: outputsbomdiff --sbom cyclonedx syft_container-sbom_cyclonedx.json trivy_container-sbom_cyclonedx.json: outputApplication (
requirements.txt) SBOM GenerationNote: We are only only looking at Build and Source SBOMs (reference), thus any tool that cannot take a
requirements.txtfile (or similar) is disqualified from the selection process.Tools Evaluated
syft requirements.txt -o cyclonedx-json \| jq > syft_application-sbom_cyclonedx.jsonjqfor improved readabilitysyft requirements.txt -o spdx-json \| jq > syft_application-sbom_spdx.jsonjqfor improved readabilitytrivy fs --format cyclonedx --output trivy_application-sbom_cyclonedx.json requirements.txttrivy fs --format spdx-json --output trivy_application-sbom_spdx.json requirements.txt--formatneeds to bespdx-jsonto be JSON, whereascyclonedxgenerates JSONcyclonedx-py requirements requirements.txt > cyclonedx-python_application-sbom_cyclonedx.jsonsbom4python -r requirements.txt --sbom spdx --format json -o sbom4python_application-sbom_spdx.jsonsbom4python -r requirements.txt --sbom cyclonedx --format json -o sbom4python_application-sbom_cyclonedx.jsonMerging Tool
Tools Evaluated
Annotation tool
TODO
Conformance Check
We want all our SBOMs to meet National Telecommunications and Information Administration (NTIA)'s minimum elements and have this checked automatically in the CI/CD pipeline.