chainloop-dev / chainloop

Chainloop is an Open Source evidence store for your Software Supply Chain attestations, SBOMs, VEX, SARIF, CSAF files, QA reports, and more.
https://docs.chainloop.dev
Apache License 2.0
365 stars 27 forks source link

feat: attestation usability improvements #934

Open migmartri opened 3 months ago

migmartri commented 3 months ago

I have a couple of usability suggestions that might be worth implementing

wdyt?

### Tasks
- [ ] explain discovery mode resolution when falling back
- [ ] output error when it's not part of autodiscovery
- [ ] https://github.com/chainloop-dev/chainloop/issues/911
- [ ] Document attestation modes https://github.com/chainloop-dev/chainloop/issues/901
- [ ] Look into accepting invalid artifacts
javirln commented 3 months ago

Regarding the last topic, here's the task for it with in depth explanation: https://github.com/chainloop-dev/chainloop/issues/911

jiparis commented 3 months ago

Thanks @migmartri. I'm afraid malformed documents are going to be a frequent issue. Given the fact that Chainloop's role is attesting, it might be reasonable to allow skipping the validation and still store it as a SBOM_CYCLONEDX_JSON, probably with an annotation validated=false. This way, Chainloop is attesting that the document exists but it's not validated so it cannot be fully trusted.

anoncam commented 3 months ago

Thanks @migmartri. I'm afraid malformed documents are going to be a frequent issue. Given the fact that Chainloop's role is attesting, it might be reasonable to allow skipping the validation and still store it as a SBOM_CYCLONEDX_JSON, probably with an annotation validated=false. This way, Chainloop is attesting that the document exists but it's not validated so it cannot be fully trusted.

Completely agree here. If a validation skip feature is implemented I think it should be explicit (user has to declare that is the desired outcome) and then that must be reflected in the metadata. My two cents.

migmartri commented 3 months ago

Thanks for the feedback. I agree with you both. What I am not sure about is if this feature should be either.

I am leaning towards b since it can inform us whether the document was valid. If we skip the validation, we lose that info.

In other words, I'd implement something like att add --soft-validation to add the evidence regardless of the validation output. The result will contain two extra annotations

These two annotations would allow users for example to create a policy that enforces not to use this feature, meaning soft-validation can't be true.

Wdyt?