Talos cluster fails to start

[tmojzes]

/kind bug

What steps did you take and what happened:

I am trying to set up a k8s cluster on Hetzner but it fails.

Commands used for cluster setup:

Local bootstrap cluster setup:

k3d cluster create --config k3d_config.yaml

• k3d_config.yaml:

  apiVersion: k3d.io/v1alpha5
  kind: Simple
  metadata:
  name: cluster-api-bootstrap
  servers: 1
  agents: 3
  kubeAPI: # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1)
  host: "k3d.localhost" # important for the `server` setting in the kubeconfig
  hostIP: "127.0.0.1" # where the Kubernetes API will be listening on
  hostPort: "6445" # where the Kubernetes API listening port will be mapped to on your host system
  image: rancher/k3s:v1.27.4-k3s1
  ports:
  - port: 8888:80 # same as `--port '8888:80@loadbalancer'`
  nodeFilters:
    - loadbalancer
  registries: # define how registries should be created or used
  create: # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost`
  name: registry.localhost
  host: "localhost"
  hostPort: "5005"
  volumes:
    - /home/<user>/registries/cluster_api_bootstrap:/var/lib/registry # persist registry data locally
  config: | # tell K3s to use this registry when pulling from DockerHub
  mirrors:
    docker-hub.localhost:
      endpoint:
        - https://registry-1.docker.io
    registry-k8s.localhost:
      endpoint:
        - https://registry.k8s.io
    gcr.localhost:
      endpoint:
        - https://gcr.io
    github-registry.localhost:
      endpoint:
        - https://ghcr.io
  configs:
    docker-hub.localhost:
      auth:
        username: "<user>" 
        password: "<password>"

  Commands used for creating a Talos cluster on Hetzner:

  
  clusterctl init --bootstrap talos --control-plane talos --infrastructure hetzner
  clusterctl generate cluster talos-cluster --kubernetes-version v1.27.4 --control-plane-machine-count=1 --worker-machine-count=1 --flavor=hcloud-talos-packer > talos-cluster.yaml

kubectl create secret generic hetzner --from-literal=hcloud=$HCLOUD_TOKEN --from-literal=robot-user=$HETZNER_ROBOT_USER --from-literal=robot-password=$HETZNER_ROBOT_PASSWORD

kubectl create secret generic robot-ssh --from-literal=sshkey-name=cluster --from-file=ssh-privatekey=$HETZNER_SSH_PRIV_PATH --from-file=ssh-publickey=$HETZNER_SSH_PUB_PATH

Patch the created secrets so they are automatically moved to the target cluster later.

kubectl patch secret hetzner -p '{"metadata":{"labels":{"clusterctl.cluster.x-k8s.io/move":""}}}' kubectl patch secret robot-ssh -p '{"metadata":{"labels":{"clusterctl.cluster.x-k8s.io/move":""}}}' kubectl apply -f talos-cluster.yaml

**Capi contronller version: 1.5.0**
**Logs:**
```bash
2023/08/25 12:47:59 http: TLS handshake error from 10.42.3.0:48038: EOF
2023/08/25 12:47:59 http: TLS handshake error from 10.42.3.0:48052: EOF
E0825 12:48:06.960220       1 controller.go:324] "Reconciler error" err="failed to create cluster accessor: error creating client for remote cluster \"default/talos-cluster\": error getting rest mapping: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: client rate limiter Wait returned an error: context deadline exceeded - error from a previous attempt: EOF" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/talos-cluster-md-0-677674857cxxnncw-hs9lq" namespace="default" name="talos-cluster-md-0-677674857cxxnncw-hs9lq" reconcileID=bfad2f24-36d3-4425-b7ba-cce46b8374c6
I0825 12:48:06.960701       1 machine_controller_phases.go:280] "Infrastructure provider has completed machine infrastructure provisioning and reports status.ready" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/talos-cluster-md-0-677674857cxxnncw-hs9lq" namespace="default" name="talos-cluster-md-0-677674857cxxnncw-hs9lq" reconcileID=023436fa-7eda-44bb-99aa-24fb95dd75c9 MachineSet="default/talos-cluster-md-0-677674857cxxnncw" MachineDeployment="default/talos-cluster-md-0" Cluster="default/talos-cluster" HCloudMachine="default/talos-cluster-md-0-mjn5z"
2023/08/25 12:48:06 http: TLS handshake error from 10.42.3.0:46958: EOF

Caph controller version: v1.0.0-beta.19 Logs:

{"level":"INFO","time":"2023-08-25T12:46:44.877Z","file":"controllers/hetznercluster_controller.go:110","message":"Cluster Controller has not yet set OwnerRef","controller":"hetznercluster","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"HetznerCluster","HetznerCluster":{"name":"talos-cluster","namespace":"default"},"namespace":"default","name":"talos-cluster","reconcileID":"2040d212-6038-49e8-b4fe-6d309c556c2c","HetznerCluster":{"name":"talos-cluster","namespace":"default"},"Cluster":{"name":""}}
{"level":"INFO","time":"2023-08-25T12:46:48.535Z","file":"controllers/hcloudmachinetemplate_controller.go:66","message":"HCloudMachineTemplate is missing cluster label or cluster does not exist default/talos-cluster-control-plane","controller":"hcloudmachinetemplate","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"HCloudMachineTemplate","HCloudMachineTemplate":{"name":"talos-cluster-control-plane","namespace":"default"},"namespace":"default","name":"talos-cluster-control-plane","reconcileID":"46a5bb12-90b2-4098-a20c-6f97278297a2","HCloudMachineTemplate":{"name":"talos-cluster-control-plane","namespace":"default"}

Cabpt controller version: v0.6.1 Logs:

2023-08-25T13:01:49Z    INFO    controllers.TalosConfig.cabpt-controller.namespace=default.talosconfig=talos-cluster-control-plane-6x28h.owner-name=talos-cluster-control-plane-jwrj5   updating talosconfig    {"endpoints": ["168.119.55.198", "2a01:4f8:c012:1c95::1"], "secret": "talos-cluster-talosconfig"}
2023-08-25T13:01:59Z    INFO    controllers.TalosConfig.cabpt-controller.namespace=default.talosconfig=talos-cluster-control-plane-6x28h.owner-name=talos-cluster-control-plane-jwrj5   ignoring an already ready config
2023-08-25T13:01:59Z    INFO    controllers.TalosConfig.cabpt-controller.namespace=default.talosconfig=talos-cluster-control-plane-6x28h.owner-name=talos-cluster-control-plane-jwrj5   updating talosconfig    {"endpoints": ["168.119.55.198", "2a01:4f8:c012:1c95::1"], "secret": "talos-cluster-talosconfig"}
2023-08-25T13:04:50Z    INFO    controllers.TalosConfig.cabpt-controller.namespace=default.talosconfig=talos-cluster-control-plane-6x28h.owner-name=talos-cluster-control-plane-jwrj5   ignoring an already ready config
2023-08-25T13:04:50Z    INFO    controllers.TalosConfig.cabpt-controller.namespace=default.talosconfig=talos-cluster-control-plane-6x28h.owner-name=talos-cluster-control-plane-jwrj5   updating talosconfig    {"endpoints": ["168.119.55.198", "2a01:4f8:c012:1c95::1"], "secret": "talos-cluster-talosconfig"}

Cacppt version: v0.5.2 Logs:

2023-08-25T13:08:28Z    INFO    reconcile TalosControlPlane {"controller": "taloscontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "TalosControlPlane", "TalosControlPlane": {"name":"talos-cluster-control-plane","namespace":"default"}, "namespace": "default", "name": "talos-cluster-control-plane", "reconcileID": "96fb04ed-8e3a-446e-8b85-74096f9bce67", "cluster": "talos-cluster"}
2023-08-25T13:08:28Z    INFO    controllers.TalosControlPlane   verifying etcd health on all nodes  {"node": "talos-cluster-control-plane-jwrj5"}
2023-08-25T13:08:28Z    INFO    controllers.TalosControlPlane   attempting to set control plane status
2023-08-25T13:08:39Z    INFO    controllers.TalosControlPlane   failed to get kubeconfig for the cluster    {"error": "failed to create cluster accessor: error creating client for remote cluster \"default/talos-cluster\": error getting rest mapping: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: client rate limiter Wait returned an error: context deadline exceeded - error from a previous attempt: EOF", "errorVerbose": "failed to get API group resources: unable to retrieve the complete list of server APIs: v1: client rate limiter Wait returned an error: context deadline exceeded - error from a previous attempt: EOF\nerror creating client for remote cluster \"default/talos-cluster\": error getting rest mapping\nsigs.k8s.io/cluster-api/controllers/remote.(*ClusterCacheTracker).createClient\n\t/.cache/mod/sigs.k8s.io/cluster-api@v1.5.0/controllers/remote/cluster_cache_tracker.go:396\nsigs.k8s.io/cluster-api/controllers/remote.(*ClusterCacheTracker).newClusterAccessor\n\t/.cache/mod/sigs.k8s.io/cluster-api@v1.5.0/controllers/remote/cluster_cache_tracker.go:299\nsigs.k8s.io/cluster-api/controllers/remote.(*ClusterCacheTracker).getClusterAccessor\n\t/.cache/mod/sigs.k8s.io/cluster-api@v1.5.0/controllers/remote/cluster_cache_tracker.go:273\nsigs.k8s.io/cluster-api/controllers/remote.(*ClusterCacheTracker).GetClient\n\t/.cache/mod/sigs.k8s.io/cluster-api@v1.5.0/controllers/remote/cluster_cache_tracker.go:180\ngithub.com/siderolabs/cluster-api-control-plane-provider-talos/controllers.(*TalosControlPlaneReconciler).updateStatus\n\t/src/controllers/taloscontrolplane_controller.go:562\ngithub.com/siderolabs/cluster-api-control-plane-provider-talos/controllers.(*TalosControlPlaneReconciler).Reconcile.func1\n\t/src/controllers/taloscontrolplane_controller.go:155\ngithub.com/siderolabs/cluster-api-control-plane-provider-talos/controllers.(*TalosControlPlaneReconciler).Reconcile\n\t/src/controllers/taloscontrolplane_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/toolchain/go/src/runtime/asm_amd64.s:1598\nfailed to create cluster accessor\nsigs.k8s.io/cluster-api/controllers/remote.(*ClusterCacheTracker).getClusterAccessor\n\t/.cache/mod/sigs.k8s.io/cluster-api@v1.5.0/controllers/remote/cluster_cache_tracker.go:275\nsigs.k8s.io/cluster-api/controllers/remote.(*ClusterCacheTracker).GetClient\n\t/.cache/mod/sigs.k8s.io/cluster-api@v1.5.0/controllers/remote/cluster_cache_tracker.go:180\ngithub.com/siderolabs/cluster-api-control-plane-provider-talos/controllers.(*TalosControlPlaneReconciler).updateStatus\n\t/src/controllers/taloscontrolplane_controller.go:562\ngithub.com/siderolabs/cluster-api-control-plane-provider-talos/controllers.(*TalosControlPlaneReconciler).Reconcile.func1\n\t/src/controllers/taloscontrolplane_controller.go:155\ngithub.com/siderolabs/cluster-api-control-plane-provider-talos/controllers.(*TalosControlPlaneReconciler).Reconcile\n\t/src/controllers/taloscontrolplane_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/.cache/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/toolchain/go/src/runtime/asm_amd64.s:1598"}
2023-08-25T13:08:39Z    INFO    controllers.TalosControlPlane   successfully updated control plane status   {"namespace": "default", "talosControlPlane": "talos-cluster-control-plane", "cluster": "talos-cluster"}
2023-08-25T13:08:39Z    INFO    reconcile TalosControlPlane {"controller": "taloscontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "TalosControlPlane", "TalosControlPlane": {"name":"talos-cluster-control-plane","namespace":"default"}, "namespace": "default", "name": "talos-cluster-control-plane", "reconcileID": "37eba9e8-0f3b-4958-9a42-443fb9cb57d0", "cluster": "talos-cluster"}
2023-08-25T13:08:39Z    INFO    controllers.TalosControlPlane   verifying etcd health on all nodes  {"node": "talos-cluster-control-plane-jwrj5"}
2023-08-25T13:08:39Z    INFO    controllers.TalosControlPlane   attempting to set control plane status

Talos logs:

What did you expect to happen: A working cluster that can be reached with kubectl and talosctl on the loadbalancer's public IP.

Environment:

• cluster-api-provider-hetzner version: caph:v1.0.0-beta.19

• Kubernetes version: Client Version: v1.28.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.4+k3s1

• OS: Pop!_OS 22.04 LTS

[lieberlois]

@tmojzes Did you get this running?

[tmojzes]

@lieberlois Unfortunately not, I have tried today with the latest version of the providers but failed like before. Have you tried it yourself?

[lieberlois]

@tmojzes Tried it aswell, didnt work. I also didnt get other bootstrap providers (k3s in my case) running with this hetzner infrastructure provider

[guettli]

Unofficial feedback from me (Syself employee): We currently see no benefit in supporting Talos. I personally like it, but overall we are happy with kubeadm and debian/ubuntu based images.