What to do with ambiguous results?

[christopher-henderson]

We have lints that that may apply or that may pass/fail, however the runtime does not have quite enough information to prove it.

An example is in #491 wherein a certificate would fail if-and-only-if one of its parents has the TLS auth EKU. We have a discussion in #428 on enabling operators to include chains into runs of the tool in order to disambiguate such cases. However, what if the operator does not include the chain? How do we let them know that there is a lint that might benefit from seeing the full chain (as simply hiding it in the docs is a great way for these sorts of lints to go unnoticed)?

And certificate chains aside, I am sure that there will be (are?) additional lints that are otherwise ambiguous (perhaps they need access to some CSV or database that's not always going to be there, for example certificate transparency records).

Some Questions

• How "loud" are these notices of ambiguous runs? Is an ambiguous Error just as loud as a proven Error? That seems noisy, however Errors are tied to MUSTS - if I were banking on this tool for my job I would personally appreciate it being this strict. Maybe --strict/--nostrict type flag (let's say that by default an unproven Error results in a non-zero exit code and you can disable that behavior via --nostrict)?
• Ambiguous Warnings, however? At first blush, these actually seem even less important than an actually Warning.

[cardonator]

Another scenario where the ambiguity is difficult is anywhere in the BRs where compliant participants MUST have a process for X or so forth, aka areas where no additional context would be able to help zlint provide a more concrete determination.

For context, that was the issue leading to #493 in which the BRs state (3.2.2.6):

the CA MUST establish and follow a documented procedure[^pubsuffix] that determines if the wildcard character
occurs in the first label position to the left of a “registry‐controlled” label or “public suffix”

[sleevi]

@cardonator So for those lints, I'm taking it as you'd support a CookieMonster level, because just like cookies are a sometime food, these are certificates that are sometimes good? 😜

[cardonator]

@sleevi haha! I was thinking it's more like classic CookieMonster where you get all the cookies you want unless you're being a bad boy. 😉

[sleevi]

Jokes aside, on a pragmatic level, I would think that for situations like that - where something is bad UNLESS the CA has taken steps X/Y/Z (such as a documented process) - it probably makes sense to assume the negative case entirely, and then allow the CA to suppress certain lints on a case-by-case basis (e.g. if they have a documented process).

The downside to this approach is the common downside we see throughout the CABF, which is that if the requirements change in, say, v1.x.y to 1.y.z, and we reuse the same lint, the CA may have disabled the lint for the new requirement. However, since a new requirement generally means a new effective date, and a new effective date generally means a new lint, I suspect the liklihood is that a new lint for the new requirement(s) would be introduced, and the CA would squelch the new lint if it doesn't apply.

[cardonator]

I like that approach, it's kind of like an attestation for specific requirements that are Error scenarios without an attestation.

[sleevi]

Exactly. Whether that attestation is implicit (i.e. the CA disabled the lint) or explicit (e.g. a "ZLint config" which can contain all of the relevant attestations a CA is making) I'm somewhat ambivalent to. I suspect that either way, there will be plenty of ways for things to go wrong, so I lean on the KISS and leave the work up to the CA to manage those attestations themselves, without any overly complex guardrails.