Closed maelvls closed 1 year ago
We found an issue on the Venafi TLS Protect DC side. We are checking if there is an acceptable workaround which can be used by vCert SDK.
I'll change the title to make it more suitable. Current title prints what our message when unmatching certificate and the private key are returned by any platform. We expect this error if something is wrong when cert key-pair is returned, and having it as a title doesn't highlight what the actual issue is.
Addressed in revert done in release v4.24.0
Just to clarify, cert-manager 1.11 and 1.12 are not currently using the upstream version VCert. cert-manager 1.11 and 1.12 is relying on a fork (jetstack/vcert) that contains this fix that adds the "reset" operation as part of cert-manager's Venafi issuances.
Note that this fork is no longer used since cert-manager 1.13.
Here is the release note that was published in cert-manager 1.11.1 and 1.12.0 and in venafi-enhanced-issuer v0.3.1:
The auto-retry mechanism added in VCert 4.23.0 and part of cert-manager 1.11.0 (https://github.com/cert-manager/cert-manager/pull/5674) and venafi-enhanced-issuer v0.2.0 and v0.3.0 has been found to be faulty.
Until this issue is fixed upstream, we now use a patched version of VCert. This patch will slowdown the issuance of certificates by 9% in case of heavy load on TPP. We aim to release at an ulterior date a patch release of cert-manager to fix this slowdown.
When using VCert v4.23.0, with TPP 22.4.1 (also tested with TPP 20.4.0), I often receive the following error message when requesting a certificate:
I checked that the problem does not come from re-using the private key. I can confirm that the CSR and the issued certificate are mismatched.
The bug
Affected: TPP 22.4.1 and older, VCert 4.23.0, cert-manager 1.11.0 (just this one version), venafi-enhanced-issuer v0.2.0 and v0.3.0.
Fixed In: VCert v4.24.0, cert-manager v1.11.1 and 1.12.0, venafi-enhanced-issuer v0.3.1.
This bug systematically happens given the following circumstances:
In real-world usage, that means that the workaround for "Click retry" introduced in #269 only "works" 50% of the time. This is better than before #269, since you were getting stuck with "Click retry" 100% of the time, but the error is now less descriptive.
Workaround: renew the certificate once again (given that this third attempt succeeds; otherwise, the fourth attempt will also fail, and so on).
When this bug occurs, VCert and cert-manager will show the following message:
Workaround: re-renew the certificate (given that this third attempt succeeds; otherwise, the fourth attempt will also fail, and so on).
Problem
This unexpected behavior seems to happen when request and reset(restart=true) are called back to back. When that happens, TPP gives VCert an old certificate instead of returning a 500 error.
There seems to be a bad interaction between request and reset(restart=true). Note that the
request
we make in VCert are asynchronous (WorkToDoTimeout=0), and request never return 500s. Onlyretrieve
calls may return a 500.The following flow triggers the problem:
Even with a 1 second pause between requesting and resetting, the problem still occurs:
We found that waiting for 5 seconds allows you to work around the problem. We also found that using reset(restart=false) before requesting doesn't trigger the problem.
Reproducing "unmatched key modulus" with
vcert
First, set your .envrc:
Then, get a token:
Then, make sure that the certificate doesn't already exist:
Then, make sure that the AD CS service (i.e., Microsoft Certification Authority) is running:
Then, enroll a certificate. It should succeed:
Then, turn the CA off:
Then, enroll. It should show an error:
If you do it again, you will get the key modulus error:
vcert-request-reset-incorrect.har.zip
If you do it a third time, it will show the correct 500 error, since
reset
won't be called:If you call it a fourth time, it will show "unmatched key modulus" again, and so on and so forth.
Reproducing with
curl
❌ Renewal of OK certificate. CA is down. Flow: request > reset(restart=true)
Occurence of this scenario in VCert: 100% of the time given the following circumstance:
There is an easy workaround: re-renewing the certificate (given that this third attempt succeeds; otherwise, the fourth attempt will also fail, and so on).
Before running this test, I turn the CA on, issue a certificate, and turn the CA off:
The actual test:
For some reason, the retrieve call returns 200 OK instead of 500 Internal Server Error, and the returned certificate doesn't match the CSR. The certificate corresponds to the old certificate that was meant to be renewed:
✅ Renewal of OK certificate. CA is down. Flow: request > wait 5s > reset(restart=true)
Before running this test, I turn the CA on, issue a certificate, and turn the CA off:
Here is the actual test:
It errors as expected: