pki: can't issue a valid cross-signed certificate if vault was too old (SKID calculation)

[dmitriy-moiseev]

Describe the bug If you issue an intermediate certificate with vault version <1.10, then upgrade vault to >1.11 to create another cross-signed intermediate (as said in https://learn.hashicorp.com/tutorials/vault/pki-engine#step-8-create-a-cross-signed-intermediate) using old one's key as key_ref - old leaf certificates won't verify using new cross-signed intermediate (as they should), as SKID will differ between them (but pubkey will be the same)

To Reproduce Steps to reproduce the behavior:

0. Start vault with version <1.10
1. Create root pki vault mount -path=pki -max-lease-ttl=87600h pki
2. Generate cert vault write pki/root/generate/internal common_name="Root CA" ttl=87600h key_bits=4096
3. Create intermediate pki vault secrets enable -path=pki_int -max-lease-ttl=43800h pki
4. Generate CSR vault write pki_int/intermediate/generate/internal common_name="Int CA" ttl=43800h key_bits=4096 | jq -r '.data.csr' | tee csr_int_old.csr
5. Sign it using root ca vault write -format=json pki/root/sign-intermediate csr=@csr_int_old.csr common_name="Int CA" ttl=43800h | jq -r '.data.certificate' | tee ca_int_old.crt
6. Import it vault write pki_int/intermediate/set-signed certificate=@ca_int_old.crt
7. Create a role vault write pki_int/roles/test key_bits=2048 max_ttl=8760h client_flag=true key_usage="DigitalSignature", ou="Test" organization="Test" country="TT" require_cn=true enforce_hostnames=false allow_glob_domains=true allow_bare_domains=true server_flag=true allow_any_name=true
8. Issue a certificate vault write -format=json pki_int/issue/test common_name='Leaf' ttl=100h format=pem | tee >( jq -r '.data.certificate' > leaf_cert_old.crt)
9. Stop vault, upgrade to >1.11, start vault
10. Rotate root pki vault write -format=json pki/root/rotate/internal common_name="Root CA" issuer_name="root-2052" ttl=262800h key_bits=4096
11. Issue cross-signed csr, using old intermediate ca as key_ref KEY_ID=$(vault read -format=json pki_int/issuer/<uuid> | jq -r ".data.key_id") vault write -format=json pki_int/intermediate/cross-sign common_name="Int CA" ttl=262700h key_ref="${KEY_ID}" | jq -r '.data.csr' | tee csr_int_new.csr
12. Sign it using new root ca vault write -format=json pki/issuer/root-2052/sign-intermediate common_name="Int CA" ttl=262600h csr=@csr_int_new.csr | jq -r '.data.certificate' | tee ca_int_new.crt
13. (optional) Import it vault write pki_int/intermediate/set-signed certificate=@ca_int_new.crt
14. (optional) Set new default for pki_int (or issue using :issuer) vault write pki_int/root/replace default=<new_uuid>
15. (optional) Issue new leaf cert vault write -format=json pki_int/issue/test common_name='Leaf' ttl=100h format=pem | tee >( jq -r '.data.certificate' > leaf_cert_new.crt)
16. Verify old leaf with new ca - it will fail openssl verify -verbose -partial_chain -CAfile ca_int_new.crt leaf_cert_old.crt (you can export root ca for full bundle - it doesn't matter in this case)

Expected behavior Old leaf certificate can be verified using new cross-signed CA

Environment: Old vault was version 1.0.0, updated it to 1.11.0

Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           9
Threshold              2
Version                1.11.0
Build Date             2022-06-17T15:48:44Z
Storage Type           etcd

Vault server configuration file(s):

Additional context Version 1.10 changed GetSubjKeyID function (and added getSubjectKeyID), so we had to rollback it for us to issue a vaild certificate - https://github.com/hashicorp/vault/compare/main...melazyk:vault:vault-skid-patch

[cipherboy]

@sgmiller Sounds like we should've made the SKID change configurable. I think OpenSSL was the one that was breaking validation for me with mismatched SKIDs when I was trying it earlier with chain building; NSS and ~JDK~ Go seemed to handle it correctly (I didn't test JDK).

I believe cross-signing to an existing intermediate is the only place this really matters; AKID isn't calculated by copied. Maybe we should support a SKID value on sign-intermediate explicitly for cross-signing? That'd solve this problem without needing the global config flag logic and let you cross-sign inside Vault to an intermediate originally generated outside of Vault (or with an older version).

(New certs don't matter and leaves don't matter as they're effectively two different certs except in rare cases where you hold long-term signatures with the same key material and still need SKID on the leaves to match for some reason).

[sgmiller]

Yeah, I like the extra sign flag, nice and simple.

[nehatomar12]

@sgmiller @cipherboy I am using vault version v1.12.2

I am trying to rotate CA, using steps: https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine#step-8-create-a-cross-signed-intermediate

but when I try to verify Old leaf certificate using new cross-signed CA, its give error

openssl verify -CAfile inter_ca.crt request.crt
CN = XXXXX
error 20 at 0 depth lookup: unable to get local issuer certificate
error request.crt: verification failed

any idea what's wrong here

[cipherboy]

@nehatomar12 The error message above doesn't contain enough information to identify that. Could you provide copies of the working and broken CA chain?

This likely belongs as a separate ticket, either here or filed through support.

[nehatomar12]

@cipherboy what SKID's I have to pass when creating cross-signed intermediate can you share the example? I don't see any SKID field in steps https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine#step-8-create-a-cross-signed-intermediate

[cipherboy]

You would pass the SKID of the existing intermediate.

This likely belongs as a separate ticket, either here or filed through support.