search

vadimkim / cert-manager-webhook-hetzner

cert-manager webhook for Hetzner DNS API
https://dns.hetzner.com/api-docs
Apache License 2.0
95 stars 39 forks source link

Logs flooded with "Unable to authenticate the request" #72

Closed dimw closed 1 month ago

dimw commented 1 month ago

I'm migrating from mecodia/cert-manager-webhook-hetzner to this project on a Microk8s cluster. After removing the old certificate manager and configuring the new one as described in the documentation, I spotted that the CertificateRequest for the wildcard certificate stuck in the pending state having the following message:

Waiting on certificate issuance from order cert-manager/[redacted]-wildcard-cert-62rtw-3600047928: "pending"

Digging deeper, I observed that cert-manager-webhook-hetzner is fooding the logs with "Unable to authenticate the request" errors producing ~10 of these per second:

$ kubectl logs --namespace cert-manager cert-manager-webhook-hetzner-85d8cf5df7-tmgzh

I0824 11:08:07.912858       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0824 11:08:07.912875       1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authe ntication::client-ca-file
I0824 11:08:07.913013       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0824 11:08:07.913037       1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
W0824 11:08:07.914160       1 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta3 PriorityLevelConfiguration is deprecated in v1.29+, unavailable in v1.32+; use flowcontrol.apiserver.k8s.io/v1 PriorityLevelConfiguration
W0824 11:08:07.916215       1 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema is deprecated in v1.29+, unavailable in v1.32+; use flowcontrol.apiserver.k8s.io/v1 FlowSchema
W0824 11:08:07.918734       1 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta3 PriorityLevelConfiguration is deprecated in v1.29+, unavailable in v1.32+; use flowcontrol.apiserver.k8s.io/v1 PriorityLevelConfiguration
W0824 11:08:07.918896       1 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema is deprecated in v1.29+, unavailable in v1.32+; use flowcontrol.apiserver.k8s.io/v1 FlowSchema
I0824 11:08:08.011998       1 apf_controller.go:366] Running API Priority and Fairness config worker
I0824 11:08:08.012264       1 apf_controller.go:369] Running API Priority and Fairness periodic rebalancing process
I0824 11:08:08.013014       1 shared_informer.go:280] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0824 11:08:08.013038       1 shared_informer.go:280] Caches are synced for RequestHeaderAuthRequestController
I0824 11:08:08.013781       1 shared_informer.go:280] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0824 11:08:09.168514       1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2024-08-24T11:08:09Z is after 2024-07-06T06:57:54Z, verifying certificate SN=xxx720, SKID=, AKID=xxx:CF failed: x509: certificate has expired or is not yet valid: current time 2024-08-24T11:08:09Z is after 2024-07-06T06:57:54Z]"
E0824 11:08:09.169254       1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2024-08-24T11:08:09Z is after 2024-07-06T06:57:54Z, verifying certificate SN=xxx720, SKID=, AKID=xxx:CF failed: x509: certificate has expired or is not yet valid: current time 2024-08-24T11:08:09Z is after 2024-07-06T06:57:54Z]"
E0824 11:08:09.176080       1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2024-08-24T11:08:09Z is after 2024-07-06T06:57:54Z, verifying certificate SN=xxx720, SKID=, AKID=xxx:CF failed: x509: certificate has expired or is not yet valid: current time 2024-08-24T11:08:09Z is after 2024-07-06T06:57:54Z]"
...

It feels like an old certificate stuck somewhere in K8s which is causing that. Unfortunately, I'm not able to find the place where the respective error message is created as there is no authentication.go in the project. Maybe it's rel

I already tried to reinstall the cert-manager, cert-manager-webhook-hetzner as well as removed all certificates I could find without any success.

Do you have any ideas why the error logs are happening? Is it related to the endless "pending" state of the certificate?

dimw commented 1 month ago

Looks like I found the issue. Apparently, the front-proxy-client certificate, which seems to be used to authenticate against the K8s API, has expired and was not auto-renewed on one of the nodes of the Microk8s cluster (although a similar issue occurs with Kubernetes in general, as seen in this Stack Overflow answer: https://stackoverflow.com/a/72111095).

To diagnose the issue, I used the following command:

$ microk8s refresh-certs -c
The CA certificate will expire in 2828 days.
The server certificate will expire in 315 days.
The front proxy client certificate will expire in -50 days.

Refreshing the expired certificate helped to resolve the flooded logs of cert-manager-webhook-hetzner:

$ microk8s refresh-certs --cert front-proxy-client.crt

After resolving, I was able to see proper logs in cert-manager-webhook-hetzner and the wildcard certificate was generated successfully:

Certificate fetched from issuer successfully

It would be great to have a more verbose log in cert-manager for easier debugging. However, the root cause of the issue was not related to this project.