tern-tools / tern

Tern is a software composition analysis tool and Python library that generates a Software Bill of Materials for container images and Dockerfiles. The SBOM that Tern generates will give you a layer-by-layer view of what's inside your container in a variety of formats including human-readable, JSON, HTML, SPDX and more.
BSD 2-Clause "Simplified" License
967 stars 188 forks source link

What is the best test image for integration testing? #1236

Open armintaenzertng opened 1 year ago

armintaenzertng commented 1 year ago

As I'm currently updating the SPDX output of tern, I noticed that this part of the code does not seem to be tested (please correct me if I'm wrong and overlooked something). To make sure that the old and new SPDX generators actually produce the same output, I'd like to include at least an integration test for the workflow starting from an Image and resulting in an SPDX JSON file.

Now, what would be a good test image for this endeavor? Are there any existing test images that I could use? The SPDX output I get with tern report -x scancode -i golang:1.12-alpine is 120k lines long, which sounds like overkill, but I'm not quite sure which parts of the image could be dropped while maintaining full test coverage, or if this actually covers everything.

rnjudge commented 1 year ago

@armintaenzertng For a general image that is subject to change like any other image, there's photon:3.0 which is a VMware minimal, single layer image. We also host a docker image in the "vmware" namespace that we use in tests (vmware/tern@sha256:20b32a9a20752aa1ad7582c667704fda9f004cc4bfd8601fac7f2656c7567bb4) that is just tern packaged as a container which would not change since this image is not updated. Note that there is some long term stability concerns around the tern test image as discussed here.

armintaenzertng commented 1 year ago

Thanks for the pointers, @rnjudge! :) Unfortunately, neither of these examples results in any files in the output SPDX, so we would have a blind spot there if we used them for testing. Maybe a slightly more complex example would be better?

rnjudge commented 1 year ago

File information will only happen if you use Scancode. No file output with Scancode for those images?

armintaenzertng commented 1 year ago

I used tern report -x scancode -i photon:3.0 -f spdxjson -o photon_test.json. No file information to be found in photon_test.json.