search

ratify-project / ratify

Artifact Ratification Framework
https://ratify.dev
Apache License 2.0
223 stars 62 forks source link

Configuring Ratify to trust a Certificate Authority + problems with Azure Key Vault #576

Closed noelbundick-msft closed 1 year ago

noelbundick-msft commented 1 year ago

I want to enable the following scenario:

TLDR: This works fine until I need to reference that root/intermediate CA cert via Key Vault, because you can't have a KV cert that doesn't include the private key

Below is a walkthrough that shows what works + what doesn't as of today

Infra team: Create a Certificate Authority

To demonstrate the issue, we'll put on our Infra Team hat and stand up a simple local CA. Companies do this in a myriad of different ways, so we'll keep it straightforward for this demo

# https://github.com/notaryproject/notaryproject/blob/main/specs/signature-specification.md#certificate-requirements

# define extensions to create certs that comply with notation
cp /usr/lib/ssl/openssl.cnf openssl.cnf
cat <<EOF >> openssl.cnf
[ notation_ca ]
# Extensions for a typical CA
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true
keyUsage = critical,cRLSign, keyCertSign

[ notation_cert ]
keyUsage = critical,digitalSignature
EOF

# create root CA key
openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096

# create root CA certificate
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -extensions 'notation_ca' -config openssl.cnf -out ca.crt -subj "/C=US/ST=WA/L=Redmond/O=My Company/OU=My Org/CN=ca.example.com"

Create a pipeline cert signing request

I interact with the infra team (web portal, service ticket, etc) to get a cert. Let's say in this case, I create a certificate signing request from Key Vault where the private key will stay and never be exported.

# IMPORTANT: cert chains *must* be stored in Key Vault in PEM format. PFX/PKCS12 is broken (notation-azure-kv enumerates certs in the wrong order)
# IMPORTANT: nonexportable private keys (ex: HSM) *must* use PEM format. PKCS12 support in Golang fails parsing when there is no private key. This will not be fixed.
POLICY=$(cat <<EOF
{
  "issuerParameters": {
    "certificateTransparency": null,
    "name": "Unknown"
  },
  "keyProperties": {
    "exportable": false
  },
  "secretProperties": {
    "contentType": "application/x-pem-file"
  },
  "x509CertificateProperties": {
    "ekus": [
      "1.3.6.1.5.5.7.3.3"
    ],
    "keyUsage": [
      "digitalSignature"
    ],
    "subject": "C=US, ST=WA, L=Redmond, O=My Company, OU=My Org, CN=pipeline.example.com",
    "validityInMonths": 12
  }
}
EOF
)

# create a certificate signing request via keyvault
# IMPORTANT: the response doesn't contain the necessary header/footer to the certificate signing request so we add it here
echo '-----BEGIN CERTIFICATE REQUEST-----' > pipelinecert.csr
az keyvault certificate create --vault-name noeltrash1 --name pipelinecert --policy "${POLICY}" --query csr -o tsv >> pipelinecert.csr
echo '-----END CERTIFICATE REQUEST-----' >> pipelinecert.csr

Infra Team: Complete the CSR

The infra team approves, and I get back a cert that's a child of my company CA root

# complete the certificate signing request using the local CA
openssl x509 -req -in pipelinecert.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out pipelinecert.crt -days 365 -sha256 -extensions 'notation_cert' -extfile openssl.cnf

# append the CA cert to create a certificate bundle
cat ca.crt >> pipelinecert.crt

Upload the cert to Key Vault

I merge that cert into Key Vault, and now I can use it

# complete the signing request and push the cert to Key Vault
az keyvault certificate pending merge --vault-name noeltrash1 --name pipelinecert -f pipelinecert.crt

Sign an image using the leaf cert

I use it with notation to sign an image

# add the pipeline key
KEY_ID=$(az keyvault certificate show --vault-name noeltrash1 --name pipelinecert --query kid -o tsv)
notation key add pipelinecert --plugin azure-kv --id $KEY_ID

# notation login
export NOTATION_USERNAME=00000000-0000-0000-0000-000000000000
export NOTATION_PASSWORD=$(az acr login -n noelfdpo --expose-token --query accessToken -o tsv)

# sign an image
notation sign --key pipelinecert noelfdpo.azurecr.io/sample-plugin@sha256:f2c54ba629c4da68327efa4085e7e322964c6533f7c9bcfd8d69202950e20d7f

Verify the signature via notation

I validate the signature. Note that I'm using the CA root cert public key here (which I have access to), and not the leaf pipeline cert that directly created the signature.

# we trust the CA, not individual certs
notation cert add --type ca --store certs ca.crt

# verify the signature
notation verify noelfdpo.azurecr.io/sample-plugin@sha256:f2c54ba629c4da68327efa4085e7e322964c6533f7c9bcfd8d69202950e20d7f

Verify the signature via Ratify

Now, do the same thing with Ratify. I create a Ratify config with a simple trust policy that uses notation's trust store path (just saves me copying some files around)

{
  "executor": {},
  "store": {
    "version": "1.0.0",
    "plugins": [
      {
        "name": "oras"
      }
    ]
  },
  "policy": {
    "version": "1.0.0",
    "plugin": {
      "name": "configPolicy",
      "artifactVerificationPolicies": {
        "application/vnd.cncf.notary.signature": "any"
      }
    }
  },
  "verifier": {
    "version": "1.0.0",
    "plugins": [
      {
        "name": "notaryv2",
        "artifactTypes": "application/vnd.cncf.notary.signature",
        "verificationCerts": [
          "/home/vscode/.config/notation/truststore"
        ],
        "trustPolicyDoc": {
          "version": "1.0",
          "trustPolicies": [
            {
              "name": "default",
              "registryScopes": [
                "*"
              ],
              "signatureVerification": {
                "level": "strict"
              },
              "trustStores": [
                "ca:certs"
              ],
              "trustedIdentities": [
                "*"
              ]
            }
          ]
        }
      }
    ]
  }
}

Verify the image

ratify verify -s noelfdpo.azurecr.io/sample-plugin@sha256:f2c54ba629c4da68327efa4085e7e322964c6533f7c9bcfd8d69202950e20d7f

So far, so good!

At this point, we've seen that

Azure Key Vault behavior

Azure Key Vault certificates must include the private key. Here are some commands that will fail:

az keyvault certificate import --vault-name noeltrash1 --name myca --file ca.crt
# error: Private key is not specified in the specified X.509 PEM certificate content. Please specify private key in the X.509 PEM certificate content.

az keyvault key import --vault-name noeltrash1 --name myca --pem-file ca.crt
# error: Import failed: Could not deserialize key data. The data may be in an incorrect format or it may be encrypted with an unsupported algorithm.

So I can refer to a CA file, but can't refer to a CA cert in Key Vault (unless I have & am willing to upload its private key)

What's next?

So in this scenario, I can't store this CA cert in Key Vault. And in the above, we've shown that I don't actually need it. What I really need is to get the public key into my trust store. But that flow doesn't play nicely with Ratify + k8s as of today.

Q: Do we expect companies/users to be using cert chains in their software supply chain security?

I don't have a specific customer in mind right away, but I would assume the answer is "yes"?

Q: What's the "happy path" for pointing to a public key?

I can modify the Helm chart and pass in the root CA cert as a ConfigMap/Secrets, and then volume mount those files on disk. I fully expect folks on Azure to ask "Why can't I use the Key Vault feature?", and walk them through this journey as well.

Other thoughts

Doing something like expanding ratifyTestCert in Helm values into an array with one entry per public key, and then wiring everything up in the chart, would give this category of users a "better" experience. This would effectively mean that customers that have well-governed CA's (note: I personally think this pattern is reasonable and common) wouldn't use the Key Vault feature at all

We could say: don't tell Ratify to trust the root CA, but instead an intermediate CA. We do put that private key in Key Vault. This would work at a technical level (important note: even though it's not needed by notation), but asking someone to maintain a complicated CA chain just to use Ratify + Key Vault is, for me personally, unreasonable to ask someone to do.

susanshi commented 1 year ago

thanks you so much for writing this up , we really appreciate this all the context and thought here @noelbundick-msft! @toddysm , do you have additional insights to customer workflow , or any other possible workaround?

noelbundick-msft commented 1 year ago

@susanshi Looking at the CertificateStore CRD, it seems like a good option in the future might be to implement a new spec.provider option named "inline" or similar, and enable me to do something like:

apiVersion: config.ratify.deislabs.io/v1alpha1
kind: CertificateStore
metadata:
  name: my-ca-root
spec:
  # this would tell Ratify to load directly from inline values rather than calling out to an external key store
  provider: inline
  parameters:
    # this would be only the public cert (chain) in PEM format
    certificate: |
      -----BEGIN CERTIFICATE-----
      MIIFxzCCA6+gAwIBAgIUBU8GfzuQwEbUNllUTvApn20BC2swDQYJKoZIhvcNAQEL
      BQAwazELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRAwDgYDVQQHDAdSZWRtb25k
      MRMwEQYDVQQKDApNeSBDb21wYW55MQ8wDQYDVQQLDAZNeSBPcmcxFzAVBgNVBAMM
      DmNhLmV4YW1wbGUuY29tMB4XDTIzMDEyNzE5MTIzM1oXDTMzMDEyNDE5MTIzM1ow
      azELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRAwDgYDVQQHDAdSZWRtb25kMRMw
      EQYDVQQKDApNeSBDb21wYW55MQ8wDQYDVQQLDAZNeSBPcmcxFzAVBgNVBAMMDmNh
      LmV4YW1wbGUuY29tMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuHbv
      6GuyvA8Cza0wXJQfzYo90uh1rTRuSrBL6cGdvFMdZgy2UyxL8vmQs7qHHIJupbKw
      EjURPn0rmPrIHFBmmyK7kjojOQma+AjOHJowcXZy619Qar2wTkopk6+ubtwjx8PZ
      t0LdHSRK0dUrcZDINqrjWBv8KknsgVkyP9PdNsnHH58hu8Po2kZEp+V6Yj+hWJ00
      GgKdPgrE1LVReA6fzFR/RvS8XuAeWiP06EkuyLCeq9qCmo7Idm2pNRJ10B/Nbmb1
      3O9wC90ESw0CT8cBjFYt2PogjWDWJgj15BA0xDpYA7RVrZRkFQQb2JZAD76o+ruZ
      SOb6CiYHLFVE26gIT5me+OMsJYM0HIS4KHudcc4j853cw4g13nCm4CtgGKnDoUV6
      iDhvoix26NMfDuVC8MHr3AlVuXjg9ZHB9Qo4VCmcsPIP9W0SYsOc+Fp0SpzDprOb
      XVU1uTuztbCegHD34ZMWG9lCXQSgKm99IaF3K+K/Gd/s1j6cXajgRkGvheLbDIPr
      W10MA2/rJU5SB/YyO4/G6d9OOtuwdGZOCPVVF/SVkbMTs0UGF4cj3upCTLP2IuUN
      RFVp+Ug37ixNU5jLtqHuqEX+VeKzjTMGQccbdQ4O65jdefnj8NYGu5QQfGBuCdBG
      ZSTGj9BOgTki7619ek/yVFiffeioM7prwknQJB0CAwEAAaNjMGEwHQYDVR0OBBYE
      FBSoEha9qupyX8Rtf9v4Cc4YeIE2MB8GA1UdIwQYMBaAFBSoEha9qupyX8Rtf9v4
      Cc4YeIE2MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
      DQEBCwUAA4ICAQChgydMDDTrxjzEBy5zxq5hJq88aiLG5cxl9a8xfPQTS7AU6dYd
      90X8r9EOwi5ZMgfqkqNsu7g7SUpXE8BN6uoVcC0lIyVfeOK3lIv3a8XUy8os5Atf
      RyPdacCs60KYlKozO0+E+CAh0XgCWv2TXA6HKmR/XCj/p2NjIgPFEdDKfaZHkBgi
      AB/UuocxSRjkHzu/rPQkyFuaF09nHgj/6OJ0YFMd3LXVmm+zkuC/Wpdhluyk7Hys
      +CP5i0jIjR4OG83+Zbi4ikMyBrIcOWMN6sVSaJx8ACqA91zICcy5+DUOPuLCip7w
      elmRSdEFZmJysARKYV1Yy20dl+5nN3sLgj42j40PnCCnqMUhxKPi1mUr1DJgv4vk
      JLFx7N3s9zLu3fmtSRRGsnQEH991hezyaR84osIu3QArNsi+SO80Lc9p95jvJKT6
      MNY7Xo52rlzZV6Hl3uX00mTqheWPBOETBrS0RZ0wnRSsR59+rIjRm/trR4+hRl8z
      JsIx0X5jr7Bf+KRAnKveQwck0J1A6tPmgpDkd/G17nbNLKmA5CYy0E6ZqTHl1p/r
      027OjBAHnQMcP9Zr2cv4vC8EWOjmtDGSIJi2WYhRaIqCI5MDahGYS+aLrfXPaHCu
      0IMA3zXuwyaStKnYUxRomApvhjcWvrpAD1f/QxvkixYk4to7p6USOACkKQ==
      -----END CERTIFICATE-----

And then have the CertificateStoreReconciler add/remove that from the in-memory map like it does with the "azurekeyvault" provider.

I think we'll encounter more of these cases. Ex: I anticipate wanting to verify that I'm running properly signed images from mcr.microsoft.com, but I certainly won't have access to the private keys. This would give me a consistent way to specify my public certs without having to resort to Helm postrender ConfigMap/Secret volume mount hacks