feature: 'cosign verify' add flags --ca-roots and --ca-intermediates to allow multiple CA roots

[dmitris]

Description

Problem Currently you can call cosign verify --certificate-chain with a file that contains a single CA root certificate and possibly related intermediate certificates. However, in a production environment with BYO CA (no Fulcio - as in [1], for example), due to the multiregion cloud redundancy, there may be multiple CAs that issue codesigning certificate, and it is necessary to be able to pass a certificate bundle file to cosign verify - otherwise the users of cosign verify would need to manually implement the "trial-and-error" loop calling cosign verify with different CA certificates passed as --certificate-chain values until the command succeeds! Needless to say, this would not be a good user experience 😄

Proposed Solution add a new --ca-roots and --ca-intermediates optional flags for cosign verify and the related commands (verify-attestation, verify-blob, verify-blob-attestation) which would allow to pass a CA Roots certificate bundle PEM file as well as one for the Intermediate Certificates.. The --ca-roots and --ca-intermediates would be mutually exclusive with the --certificate-chain parameter - should use one or the other but not both.

$ cosign verify --help
[...]
    --ca-roots='':
    path to a bundle file of CA Root certificates in PEM format which will be needed when building the certificate
    chains for the signing certificate. Conflicts with --certificate-chain.

    --ca-intermediates='':
    path to a bundle file of CA Intermediate certificates in PEM format which will be needed when building the certificate
    chains for the signing certificate. Conflicts with --certificate-chain.

    --certificate-chain='':
    path to a list of CA certificates in PEM format which will be needed when building the certificate chain for
    the signing certificate. Must start with the parent intermediate CA certificate of the signing certificate and
    end with the root certificate. Conflicts with --ca-roots and --ca-intermediates.

Miscellaneous https://docs.sigstore.dev/system_config/custom_components/ indicates "a last resort" configuration option with the environment variable SIGSTORE_ROOT_FILE:

This specifies an out of band PEM-encoded X.509 certificate for a custom Fulcio root certificate.

However, this doesn't help to solve the stated problem - relying on the environment variable is inconvenient and fragile in a production deployment, and the documentation talks about the file containing a single certificate, not a certificate bundle with multiple CA roots.

References [1] Scaling Up Supply Chain Security: Implementing Sigstore for Seamless Container Image Signing

Below is list of issues that are related to x509 and certificates:

• https://github.com/sigstore/cosign/pull/2461#discussion_r1024644134
• https://github.com/sigstore/cosign/issues/2472
• https://github.com/sigstore/cosign/issues/2511
• https://github.com/sigstore/cosign/issues/2632
• https://github.com/sigstore/cosign/issues/2635
• https://github.com/sigstore/cosign/issues/2939

/cc @woodruffw

[dmitris]

When looking further into the implementation, I realized there is likely a problem in terms of the possible intermediate certificates. It is not clear how the intermediate certificates would be passed if there are intermediate certificates... Currently with --certificate-chain the instructions say: "Must start with the parent intermediate CA certificate of the signing certificate and end with the root certificate", therefore it is clear what to put into Roots (the last certificate in the file) and what - into Intermediates (the other certificates, if any) for x509.VerifyOptions.

Now if we make an alternative --ca-roots flag, it is not clear whether it would be only for the Roots, or whether one could also add intermediate certificates into the bundle - but in this case, how should they be separated? I think this can be solved in the following way:

• clarify that the PEM bundle should consist only of the CA roots with no intermediates.
• for the case with multiple CA roots with intermediate certificates, add the --ca-intermediates flag which would populate verifyOptions.Intermediates.

To recap, the coverage of different cases by the flags would be:

• One CA root, potentially with one or multiple corresponding intermediate certs - use --certificate-chain as currently
• Multiple CA roots, no intermediates - use the new --ca-roots flag
• Multiple CA roots, some or all with intermediate certificates - use the ca-roots and --ca-intermediates flags

I updated the issue Subject and the proposal above, as well as the (draft) PR with a --ca-roots implementation.

[woodruffw]

Thanks for the ping @dmitris!

I don't have too much to add from cosign's UX side, but one idea: rather than two new additional flags, maybe this is something that fits under a singular --trusted-root flag, which would then be given a JSON serialized form of the TrustedRoot that the Public Good Instance uses?

(This works out, since a TrustedRoot can contain one or more CertificateAuthority models, each of which then has a cert_chain member with arbitrarily many roots and intermediates.)

For context, this is the route we're going down with sigstore-python: https://github.com/sigstore/sigstore-python/issues/821 and https://github.com/sigstore/sigstore-python/issues/779. Once we support --trusted-root, we'll likely remove all of the other individual flags that are covered by it.

[dmitris]

Thanks for the comment @woodruffw!

I can see the advantages of the "full cover" single --trusted-root flag (for example, it can include the TSA certificate as well) - but besides having to implement it, we would need to start producing and distributing the certificates in the JSON "TrustedRoot" format to all the (internal) users. I'm go investigate how much effort and time it would take.

The full migration to --trusted-root and removal of --certificate-chain (some other flags as well?) would require the v3.0.0, right? Maybe we can do the following:

• implement --trusted-root and announce that is going to be the way going forward
• implement --ca-roots as a short-term / transitionary solution
• mark both the new --ca-roots as well as the existing --certificate-chain as DEPRECATED, steer users toward the --trusted-root
• with the 3.x release, remove all the parameters except --trusted-root.

[haydentherapper]

@dmitris Sorry for the delayed response. Overall I'm supportive of this change. Splitting certificate-chain into a set of roots and a set of intermediates is definitely a step in the right direction. I'm good with adding these flags and marking certificate-chain as deprecated.

To your question about opinionated order for certificate-chain, I think these new flags are an opportunity to correct this. I noted in another bug openssl's design with a set of roots and a set of untrusted chain builders. Typically the former is only self-signed CA certificates and the latter is intermediates. I would prefer the UX to have two flags.

As you noted, you could have trusted intermediates as roots of trusts. This is where I'll differ from openssl - This is a footgun if you aren't experienced with PKI. Like you noted, what happens if the roots file contains an intermediate? I'd expect this to be a common misuse. I would prefer we require that all certs in the file are self-signed roots. I'm open to relaxing this constraint if the use-case pops up to have intermediates as the roots of trust, but we'll deal with that if/when someone asks.

I agree with @woodruffw that long-term, we want to support the TrustedRoot file to align with other Sigstore clients. I also recognize this is going to be a larger change, so delaying this to be a part of the longer-term UX changes to Cosign is fine. If you wanted to take on implementing support for this as part of this work, happy to chat more. Once this change lands, along with support for the newer bundle format, we'll release 3.0 and remove the multitude of ways to specify a chain.

Also @TomHennen since we've chatted about this, thoughts?

[TomHennen]

Letting intermediates act as roots, definitely isn't what I expected!

Here's a test where I try to cover the existing 'offline' uses and where I'm surprised that I can leave the root out and it still works. https://github.com/sigstore/cosign/commit/75485e85a494014a70f0dc5a1124c8cd9aec0817#diff-29007b577f66b9fca944d7d87e89d87a1027754f251a7e41feac1e61809b6cc4R663

To some extent I think just fixing up the command line options like you're doing would solve this?

I don't have an opinion on if it would be good/bad to differ fromm what openssl does.