Load Balancer on ingress goes to unused servers

[jaumecornado]

I have a 7 node kubernetes cluster with nginx-ingress.

If ingress is not deployed on all nodes; Load Balancer goes to mixed health. Can we go on green?

[blafasel42]

same here. IMHO LB should not get targets for nodes which are not workers...

[sergeyshevch]

Or as an alternative LB can remove the target if it is unhealthy for some period. So your target will be removed from LB

[Shadowghost]

You can manually exclude nodes from the LB until (if ever) this is automated: https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/182#issuecomment-830577642

[RyanSquared]

Just ran into this issue where we have 25+ nodes in a cluster (breaking the Hetzner load balancer limit) and the Service was pointing to all nodes even though some nodes can't even run the LoadBalancer'd application. Would be very nice to dynamically update the LoadBalancer based on whether or not a node has a Pod that is running the LoadBalancer'd application.

[sergeyshevch]

@RyanSquared i guess you can exclude this node with annotations from previous comment

[RyanSquared]

@RyanSquared i guess you can exclude this node with annotations from previous comment

We have 23 nodes that we're considering "general purpose"/"untainted", and at some point I think we're gonna have to dedicate some nodes specifically for load balancer services and have every node labeled as excluded.

[Shadowghost]

This is how Kubernetes handles it right now.

Since Kubernetes 1.21 the ServiceNodeExclusion FeatureGate is enabled by default, so you should only need to add the correct annotation to your nodes if you're on Kubernetes 1.21+

[blafasel42]

can you give us a hint, which annotation exactly, @Shadowghost?

[Shadowghost]

The one I mentioned here: https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/182#issuecomment-830577642 node.kubernetes.io/exclude-from-external-load-balancers: "true"

[blafasel42]

ah, so I annotate nodes. makes sense since controller nodes will be quite static. Thanks for the insight!

[LKaemmerling]

I will close the issue here, as the Cloud Controller Manager is not in charge of "selecting" the nodes. @Shadowghost already posted the k8s solution to exclude the nodes: https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/182#issuecomment-830577642

[easystartup-io]

CCM has support for this annotation as mentioned here

"load-balancer.hetzner.cloud/node-selector"

https://pkg.go.dev/github.com/hetznercloud/hcloud-cloud-controller-manager/internal/annotation#Name