hetznercloud / hcloud-cloud-controller-manager

Kubernetes cloud-controller-manager for Hetzner Cloud
Apache License 2.0
747 stars 116 forks source link

Cannot to disable http service creation #805

Closed DTiunov closed 1 week ago

DTiunov commented 1 week ago

TL;DR

I am trying to create Hetzner load balancer through annotations in helm file for nginx ingress controller. I have the following annotations listed:

    annotations:
      load-balancer.hetzner.cloud/name: "k8s-test-lb"
      load-balancer.hetzner.cloud/location: "fsn1"
      load-balancer.hetzner.cloud/type: "lb11"
      load-balancer.hetzner.cloud/ipv6-disabled: "true"
      load-balancer.hetzner.cloud/network-zone: "networkname"
      load-balancer.hetzner.cloud/use-private-ip: "true"
      load-balancer.hetzner.cloud/protocol: "https"
      load-balancer.hetzner.cloud/http-certificates: "certificatename"
      load-balancer.hetzner.cloud/http-redirect-http: "true"

Expected behavior

I expected that setting https in load-balancer.hetzner.cloud/protocol annotation would create only https an service (without creating http service on port 80)

Observed behavior

But I always getting two services in the created load balancer 1111 if load-balancer.hetzner.cloud/http-redirect-http is false.

And I always getting error "Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: port 80 is busy, redirect_http is enabled on another service (source_port_already_used)" if load-balancer.hetzner.cloud/http-redirect-http is true

Minimal working example

My Nginx ingress controller version is 1.11.3

NAME                            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                                     APP VERSION         
ingress-nginx-controller        kube-system     1               2024-11-25 07:20:04.093748109 +0000 UTC deployed        ingress-nginx-4.11.3                      1.11.3

And my Hetzner cloud controller manager version is 1.20.0

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                                   APP VERSION
hccm            kube-system     1               2024-10-23 23:36:31.352227602 +0000 UTC deployed        hcloud-cloud-controller-manager-1.20.0   

Log output

Full error log with "load-balancer.hetzner.cloud/http-redirect-http: true" annotation:
I1125 07:55:33.956835       1 load_balancers.go:137] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-controller-controller" nodes=["k8s-test-worker-1","k8s-test-worker-2"]
I1125 07:55:33.957067       1 event.go:389] "Event occurred" object="kube-system/ingress-nginx-controller-controller" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I1125 07:55:34.168966       1 load_balancer.go:850] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=2184461
I1125 07:55:34.941947       1 load_balancer.go:861] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=2184461
E1125 07:55:35.296514       1 controller.go:298] error processing service kube-system/ingress-nginx-controller-controller (retrying with exponential backoff): failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: port 80 is busy, redirect_http is enabled on another service (source_port_already_used)
I1125 07:55:35.296628       1 event.go:389] "Event occurred" object="kube-system/ingress-nginx-controller-controller" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBServices: port 80 is busy, redirect_http is enabled on another service (source_port_already_used)"

Additional information

Please help me, how to disable the creation of the http service?

lukasmetzner commented 1 week ago

Hi,

the nginx ingress controller is running with HTTP enabled by default, which is why the hcloud load balancer has two services. Furthermore, this also explains why the redirect can not be set up, as the port is already in use.

Deployment with default values of ingress-nginx/ingress-nginx:

$ kubectl -n ingress-nginx get svc nginx-ingress-ingress-nginx-controller -o yaml
[...]
 externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http # <---
    name: http
    nodePort: 31899
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 31079
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
[...]

You can try disabling the HTTP port in the nginx ingress controller and setting load-balancer.hetzner.cloud/http-redirect-http to true.

controller:
  service:
    annotations:
        load-balancer.hetzner.cloud/name: "k8s-test-lb"
        load-balancer.hetzner.cloud/location: "fsn1"
        load-balancer.hetzner.cloud/type: "lb11"
        load-balancer.hetzner.cloud/ipv6-disabled: "true"
        load-balancer.hetzner.cloud/network-zone: "netzone"
        load-balancer.hetzner.cloud/use-private-ip: "true"
        load-balancer.hetzner.cloud/protocol: "https"
        load-balancer.hetzner.cloud/http-certificates: "certname"
        load-balancer.hetzner.cloud/http-redirect-http: "true"
    type: LoadBalancer
    enableHttp: false

Best Regards Lukas