Load balancer being provisioned without targets

TL;DR

It all works good if I add the target to my node manually in hetzner. Otherwise, all the ports and everything are set but the target to the node. Here is the logs of the controller:

 kubectl logs -n kube-system -l app=hcloud-cloud-controller-manager --tail=100 -f

I1027 15:16:52.858565 1 event.go:307] "Event occurred" object="default/web-app" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer" I1027 15:16:52.878447 1 load_balancers.go:109] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="web-app" nodes=[] I1027 15:16:52.879167 1 event.go:307] "Event occurred" object="default/web-app" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="UnAvailableLoadBalancer" message="There are no available nodes for LoadBalancer" I1027 15:16:53.741870 1 route_controller.go:216] action for Node "node1" with CIDR "10.233.64.0/24": "keep" I1027 15:16:55.376856 1 load_balancer.go:723] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=2127367 I1027 15:16:56.411260 1 load_balancers.go:159] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=2127367 I1027 15:16:56.589854 1 event.go:307] "Event occurred" object="default/web-app" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"

Doesn't look like it's failing or anything. Here is the cluster config for the kubespray:

cloud_provider: external
external_cloud_provider: hcloud

external_hcloud_cloud:
  token_secret_name: hcloud-api-token
  with_networks: true
  service_account_name: hcloud-sa
  hcloud_api_token: "{{ lookup('env', 'HCLOUD_API_TOKEN') }}"
  controller_image_tag: v1.16.0

kube_network_plugin: cilium
network_id: kubernetes-node-network

all:
  hosts:
    node1:
      ansible_host: 172.16.0.101
      ip: 172.16.0.101
      access_ip: 172.16.0.101
  children:
    kube_control_plane:
      hosts:
        node1:
    kube_node:
      hosts:
        node1:
    etcd:
      hosts:
        node1:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}

Expected behavior

Supposed to add the internal target to the node that is running the pod.

Observed behavior

Doesn't add anything to the targets.

Minimal working example

No response

Log output

No response

Additional information

No response

Okay so the problem was that I accidentally run the kubespray setup script on the worker node, it then set the following label on it preventing loabalancer from attaching:

node.kubernetes.io/exclude-from-external-load-balancers=

root@control-plane:~# kubectl get node node1 --show-labels
NAME    STATUS   ROLES           AGE   VERSION   LABELS
node1   Ready    control-plane   14h   v1.31.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=cx22,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=hel1,failure-domain.beta.kubernetes.io/zone=hel1-dc2,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,node.kubernetes.io/instance-type=cx22,topology.kubernetes.io/region=hel1,topology.kubernetes.io/zone=hel1-dc2
root@control-plane:~#

This label explicitly tells Kubernetes to exclude this node from load balancers. Just remove it:

kubectl label node node1 node.kubernetes.io/exclude-from-external-load-balancers-

# Verify it's removed
kubectl get node node1 --show-labels

hetznercloud / hcloud-cloud-controller-manager