Add more provenance information to SecObserve

[dervoeti]

It's often hard to find out how exactly a component was brought into an image. For example, our current workflow for Java dependencies involves cloning the source repo and looking and the output of mvn dependency:tree, which is cumbersome and error prone. We want to try out tools like ScanCode or find other ways to make the information on how a component is brought in easier discoverable, SBOMs might also be a good option.

Our solution will be to generate SBOMs for all the components in our images, create "merged" SBOMs containing all the information that Syft finds on top of the information that the CycloneDX plugins generate during build. This is needed, because neither of the tools can see all the information that's neccessary: Syft doesn't see the dependency tree and the CycloneDX plugins don't resolve shaded dependencies in Java (and maybe other things too).

• [x] Create a visualization for dependency trees in SecObserve
• [x] Create a tool / workflow to merge the SBOM information
• [x] https://github.com/stackabletech/docker-images/pull/814
• [x] https://github.com/stackabletech/docker-images/pull/827
• [x] Scan our SBOMs instead of our images
• [x] Think about Kafka
• [x] Think about OPA
• [x] https://github.com/stackabletech/docker-images/pull/895

[dervoeti]

Hit some problems when generating SBOMs at build time for some products:

• Kafka: Problem with the Gradle CycloneDX plugin and Scala (https://github.com/CycloneDX/cyclonedx-gradle-plugin/issues/239)
• OPA / statsd_exporter: cyclonedx-gomod expects the source code to be in a Git repo and obtains the version from the Git tag. Solutions:
  1. Patch cyclonedx-gomod
  2. Create a fake Git repo
  3. Clone the source code from Git instead of mirroring it in Nexus

The rest looks fine so far. The products with SBOM generation problems don't have too many vulnerabilities, so for the vulnerability analysis it's not dramatic that there's no dependency tree available in SecObserve.

[NickLarsenNZ]

This has been "waiting for review" for a while now. What exactly needs reviewing?

[dervoeti]

Yeah sorry, the two PRs linked in the tasklist needed to be reviewed, which they are now.

I'll move this issue back to "Development: In progress" for now, as there are still some problems with some products (import of scan results takes a very long time in some cases, I'm debugging why exactly that's the case). After this last issue is fixed, we need to decide whether we want to put in the work to get improved SBOMs for OPA, Kafka and our operators as well. Current estimation: OPA and operators should be relatively easy, Kafka seems to be hard.

[dervoeti]

I added SBOMs for Kafka, OPA and statsd_exporter as well and improved the ones for Airflow with this PR: https://github.com/stackabletech/docker-images/pull/895

I think once this is merged, this issue can be closed.

[dervoeti]

This is done now, we have lovely SBOMs with dependency information for all our products in SecObserve. I did not implement this for our operators yet, since it would not bring any real benefit at the moment. Dependency trees for RPM packages are already present in the scan results of the operator images, and the Rust codebase in our current release does not contain any vulnerabilities. Might add it at a later point in time if it becomes useful.