[Discussion] How should we evaluate "completeness"?

[idunbarh]

We need a automated method for evaluation "completeness" of SBOMs which can be incorporated into a pipeline.

The following tools have quality checks:

• sbomqs
• sbom-utility

Ideally we be able to determine if we're meeting all minimum requirements of Framing Software Component Transparency Third Edition (DRAFT) but tooling probably doesn't support these checks.

Another option of completeness is the BOM Maturity Model which is being incorporated into sbomqs.

Are there other ways we could measure completeness of SBOMs today in a pipeline?

[surendrapathak]

sbomqs is working on supporting the draft 4 here: https://github.com/interlynk-io/sbomqs/issues/313

[idunbarh]

Lets proceed with sbomqs similar to the phase 1 - docker implementation created by @vpetersson .

  Validate:
    needs: Assemble
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Download SBOMs
        uses: actions/download-artifact@v4

      - name: Install sbomqs
        run: |
          curl -L -o /tmp/sbomqs \
            "https://github.com/interlynk-io/sbomqs/releases/download/v${SBOMQS_VERSION}/sbomqs-linux-amd64"
          chmod +x /tmp/sbomqs

      - name: "Display SBOM quality score through sbomqs"
        run: |
          echo \`\`\` >> ${GITHUB_STEP_SUMMARY}
          for SBOM in $(find . -iname *.json); do
            /tmp/sbomqs score "$SBOM" >> ${GITHUB_STEP_SUMMARY}
          done
          echo \`\`\` >> ${GITHUB_STEP_SUMMARY}

[tiegz]

Do we care about SBOM spec validation at all? I just noticed that sbomqs will only check that the version of the declared SBOM spec is supported, but it doesn't actually do validation against that spec.

There's an issue to add support here: https://github.com/interlynk-io/sbomqs/issues/248

I noticed this when sbom-utility raised some spec errors on an SBOM that were not reported by sbomqs:

[INFO] Too many errors. Showing (10/2099) errors.
1. {
        "type": "unique",
        "field": "dependencies.1.dependsOn",
        "context": "(root).dependencies.1.dependsOn",
        "description": "array items[0,1] must be unique",
        "value": {
            "type": "array",
            "index": 0,
            "item": "pkg:npm/negotiator@0.6.2"
        }
    }

[riteshnoronha]

@tiegz yes this is accurate sbomqs does not validate against the spec. This was intentional on our part. Our reasoning was as follows

• The validating against the schema, checks for every field of the sbom. While this is great, the tool, was written to check for compliance standards, like NTIA, BSI. These standards currently don't go beyond metadata/components/dependencies.
• These schemas are only available for cyclonedx, spdx nothing official exists. This will change with spdx3.0.
• The issue above is interesting, we should add that as a quality metric.

We think of validating a sbom with a schema as a pre-step to scoring. This is how we plan to implement it. It will however be optional.

-Ritesh (author sbomqs)

[idunbarh]

The group has decided to use sbomqs for the phase 1 reference implementations. I'm going to close this issue as decided.