If X509_check_ca() fails to cache X509v3 extension values, the return value may be incorrect, leading to erroneously assuming a given certificate is a CA or EE cert (while in reality it is the other, or neither).
This failure mode can arise because X509_check_ca() doesn't verify whether libcrypto's (void)x509v3_cache_extensions(x) flipped the EXFLAG_INVALID flag in x->ex_flags. Unfortunately, X509_check_ca() doesn't have a return code to indicate an error, so this can't be fixed in libcrypto - the API is broken.
The workaround is to call X509_check_purpose() with a purpose argument of -1, before calling X509_check_ca(), this ensures the X509v3 extensions are cached. Since X509_check_purpose()does have a return code to indicate errors, we can use that to supplement X509_check_ca()'s shortcomings.
OpenBSD's rpki-client also uses the above approach.
ydahhrk
commented
1 year ago
If
X509_check_ca()fails to cache X509v3 extension values, the return value may be incorrect, leading to erroneously assuming a given certificate is a CA or EE cert (while in reality it is the other, or neither).This failure mode can arise because
X509_check_ca()doesn't verify whether libcrypto's(void)x509v3_cache_extensions(x)flipped theEXFLAG_INVALIDflag inx->ex_flags. Unfortunately,X509_check_ca()doesn't have a return code to indicate an error, so this can't be fixed in libcrypto - the API is broken.The workaround is to call
X509_check_purpose()with a purpose argument of-1, before callingX509_check_ca(), this ensures the X509v3 extensions are cached. SinceX509_check_purpose()does have a return code to indicate errors, we can use that to supplementX509_check_ca()'s shortcomings.OpenBSD's rpki-client also uses the above approach.
Compare https://github.com/openssl/openssl/blob/2bcf8e69bd92e33d84c48e7d108d3d46b22f8a6d/crypto/x509v3/v3_purp.c#L84-L86 to https://github.com/openssl/openssl/blob/2bcf8e69bd92e33d84c48e7d108d3d46b22f8a6d/crypto/x509v3/v3_purp.c#L619-L620 the latter is lacking error checking.