HCCM not able to extract previosly generated (managed) TLS certificate

marcopaggioro commented 1 month ago

TL;DR

It seems that HCCM is not able to see that a certificate already exists (already created by HCCM from a re-created Service or from another Service). If it already exists then it fails and the Services in the load balancer are not produced

Expected behavior

I expect HCCM not to fail if the certificate already exists and was created by itself.

Observed behavior

When I create the Service (annotations below) for the first time I can see the new certificate in the Hetzner Certificate section

Due to that I can see that HCCM populates even the Services section of my Hetzner Load Balancer. Everything works fine

If i destroy and recrete my service, than HCCM reports these errors

E0803 16:54:50.345120       1 controller.go:298] error processing service traefik/traefik (retrying with exponential backoff): failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: hcops/hclbServiceOptsBuilder.buildAddServiceOpts: hcops/CertificateOps.GetCertificateByLabel: not found
I0803 16:54:50.345206       1 event.go:389] "Event occurred" object="traefik/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: hcops/hclbServiceOptsBuilder.buildAddServiceOpts: hcops/CertificateOps.GetCertificateByLabel: not found"

Seems like it can't detect that the certificate already exists and it fails.

If I delete the certificate nothing more happens but then If I delete and re-create the Service so HCCM "wake up" and recreated the certificate correctly (with Services in LB).

Minimal working example

Services with these annotations

  annotations:
    load-balancer.hetzner.cloud/certificate-type: managed
    load-balancer.hetzner.cloud/health-check-protocol: tcp
    load-balancer.hetzner.cloud/http-managed-certificate-domains: yourdomain.it,www.yourdomain.it,api.yourdomain.it
    load-balancer.hetzner.cloud/http-managed-certificate-name: https-certificate
    load-balancer.hetzner.cloud/http-redirect-http: 'true'
    load-balancer.hetzner.cloud/name: prod-balancer
    load-balancer.hetzner.cloud/protocol: https

Log output

E0803 16:54:50.345120       1 controller.go:298] error processing service traefik/traefik (retrying with exponential backoff): failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: hcops/hclbServiceOptsBuilder.buildAddServiceOpts: hcops/CertificateOps.GetCertificateByLabel: not found
I0803 16:54:50.345206       1 event.go:389] "Event occurred" object="traefik/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: hcops/hclbServiceOptsBuilder.buildAddServiceOpts: hcops/CertificateOps.GetCertificateByLabel: not found"



### Additional information

_No response_

apricote commented 1 month ago

The certificate has a label hcloud-ccm/service-uid that must match the kubernetes service uid. You have to update this manually right now.


$ service_uid=$(kubectl get service -n traefik traefik -o=go-template --template='{{ .metadata.uid }}')
$ echo $service_uid
$ hcloud certificate add-label --overwrite https-certificate hcloud-ccm/service-uid=$service_uid

marcopaggioro commented 1 month ago

It sounds like a good workaround but, in fact, is a workaround. No? It sounds no-sense that I have to "patch" the label of the exist certificate every time I re create the Service

apricote commented 1 month ago

Yea, we need a better way to associate the cert with the service and potentially clean it up when necessary.

apricote commented 1 month ago

Suggestions by @micheljung in #596:

HCCM would remove managed certificates when removing the load balancer (potentially undesired)
HCCM would remove the service label from the certificate when removing the load balancer
HCCM would add the service label to an existing certificate
HCCM would match the certificate by name and not by label

hetznercloud / hcloud-cloud-controller-manager