Multiple hostnames for one load-balancer not be considered

[deen13]

Hey DO-Team,

i'm currently trying to move my load-balancer configuration into my IaC. Our product exposes a http endpoint on 80/443 and a mqtt endpoint on 8883. In the user interface i was able to let my domain example.com point to my http load-balancer and my subdomain mqtt.example.com point to my mqtt load-balancer. Coming from the user interface, i'd expect that this also should be possible using kubernetes.

Kubernetes

apiVersion: v1
kind: Service
metadata:
  name: http-load-balancer
  namespace: application
  labels:
    app: frontend
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
    service.beta.kubernetes.io/do-loadbalancer-algorithm: "round_robin"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: "54005b8e-d79b-4891-8d17-286ee39ca1b4"
    service.beta.kubernetes.io/do-loadbalancer-hostname: "example.com"
spec:
  type: LoadBalancer
  selector:
    app: frontend
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
    - name: https
      protocol: TCP
      port: 443
      targetPort: 80

apiVersion: v1
kind: Service
metadata:
  name: mqtt-load-balancer
  namespace: messaging
  labels:
    app: mqtts
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-hostname: "mqtt.example.com"
spec:
  type: LoadBalancer
  selector:
    app: vernemq
  ports:
    - port: 8883
      name: mqtts

Actual Behaviour

After deploying my manifests the domain only points to the http-load-balancer.

Expected Behaviour

The domain should point to http-load-balancer, as well as to the mqtt-load-balancer.

[timoreimann]

Apologies for the super late reply -- this one fell off my radar somehow. 😔

We currently do not manage DNS records at all in CCM. The service.beta.kubernetes.io/do-loadbalancer-hostname annotation you use only causes the specified hostname to be returned in the Service status field (which can be helpful to work around an LB routing bypassing problem that's described in the documentation) but does not do any management / setup work.

What you're asking for sounds like a feature worth considering though, so I'm happy to keep the issue open for tracking purposes.

[khash]

@timoreimann not quite the same issue but related to service.beta.kubernetes.io/do-loadbalancer-hostname workaround: when using an Nginx Ingress, it is possible to use the same LB for multiple hostnames as Nginx Ingress Controller uses the Host header to find the right backend.

My understanding of the service.beta.kubernetes.io/do-loadbalancer-hostname annotation by reading the documentation is that it can only accept 1 hostname which means it wouldn't work with a single Nginx Ingress Controller for multiple addresses.

Is there a solution / workaround for this?

[timoreimann]

@khash is this about external or internal access to the LB?

I think for external access, you should be able to hit your LB with any Host header. The service.beta.kubernetes.io/do-loadbalancer-hostname annotation is meant for accessing the public LB from inside the cluster only really. If you need different Host headers for inside-to-outside routing, I can't think of a good solution since the best we have is CNAMEs but that's going to map to a single host again as far as I can see. :/

[khash]

@timoreimann this is for internal access to the LB (more specifically to pass the first step for CertManager for the CNAME before it hits LetEncrypt with the request)

[deimosfr]

Hi any news regarding that request?

[timoreimann]

Not yet.

Re-reading the issue, I'm coming to think that an approach levering Ingress might be an equivalent if not better solution to the described problem. Notably, it'd require a single LB only and provide more flexibility (with the downside of having to manage an Ingress provider and controller, though there are plenty of good options out there).

What do folks think about that?

[timoreimann]

I'm closing this one in favor of what was proposed above by me (and received some acknowledgement), which is to leverage an Ingress provider to demultiplex requests from a single Service to several downstream application.

[regnerisch]

Hey,

I'm facing the same problem with cert-manager and multiple domains on one LB. I saw this comment https://github.com/SovereignCloudStack/issues/issues/250#issuecomment-1397089228

Especially this part could be a solution for the problem?

I also investigated https://github.com/kubernetes/cloud-provider-openstack/issues/1287 mentioned by @flyersa and OpenStack Cloud Controller Manager is no exception. There is loadbalancer.openstack.org/hostname annotation, which can be used to explicitly set a hostname in the status of the load balancer service. But there is also one unique option in the occm which I have not found anywhere else. It is called enable-ingress-hostname. It is used only when there is annotation loadbalancer.openstack.org/proxy-protocol: "true". It adds dns suffix nip.io(it can be changed) to the load balancer IP address. So basically it creates hostname IPaddress.nip.io, returns it to the status of lb service and everything works.

Maybe someone could take a look into it.