[BUG] Unable to import existing generic cluster

CC007 commented 3 weeks ago

Rancher Server Setup

Rancher version: v2.9.0
Installation option (Docker install/Helm Chart): Helm Chart
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): k3s, v1.30.3+k3s1
Proxy/Cert Details: let's encrypt cert, created by rancher itself (pointed at my domain name). The certificate is valid

Information about the Cluster

Kubernetes version: v1.30.3

Cluster Type (Local/Downstream):

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Trying to import (self-hosted)

I have 2 clusters:

a 2-node Raspberry Pi cluster on which I run rancher (works perfectly)
a 8-node Raspberry Pi cluster on which I want to import into rancher

When trying to import the cluster by applying the resources to my 8-node cluster, as rancher tells me to do, the cattle-cluster-agent pod gets a CrashLoopBackoff.

To Reproduce

Set up 2-node k3s cluster using k3sup

$ kubectl get nodes -owide
NAME            STATUS   ROLES                       AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
pirack1-node1   Ready    control-plane,etcd,master   19h   v1.30.3+k3s1   192.168.1.101   <none>        Ubuntu 22.04.3 LTS   5.15.0-1059-raspi   containerd://1.7.17-k3s1
pirack2-node1   Ready    control-plane,etcd,master   19h   v1.30.3+k3s1   192.168.1.201   <none>        Ubuntu 22.04.3 LTS   5.15.0-1055-raspi   containerd://1.7.17-k3s1

Install rancher onto that cluster using these instructions: https://ranchermanager.docs.rancher.com/v2.9/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster (I chose latest, because I need Rancher v2.9. I also chose to use let's encrypt)

Set up 8-node k3s cluster using k3sup

$ kubectl get nodes -o wide
NAME            STATUS   ROLES                       AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
pirack1-node2   Ready    control-plane,etcd,master   15h   v1.30.3+k3s1   192.168.1.102   <none>        Ubuntu 22.04.3 LTS   5.15.0-1060-raspi   containerd://1.7.17-k3s1
pirack1-node3   Ready    control-plane,etcd,master   15h   v1.30.3+k3s1   192.168.1.103   <none>        Ubuntu 22.04.3 LTS   5.15.0-1059-raspi   containerd://1.7.17-k3s1
pirack1-node4   Ready    control-plane,etcd,master   15h   v1.30.3+k3s1   192.168.1.104   <none>        Ubuntu 22.04.3 LTS   5.15.0-1055-raspi   containerd://1.7.17-k3s1
pirack1-node5   Ready    <none>                      15h   v1.30.3+k3s1   192.168.1.105   <none>        Ubuntu 22.04.3 LTS   5.15.0-1055-raspi   containerd://1.7.17-k3s1
pirack2-node2   Ready    <none>                      15h   v1.30.3+k3s1   192.168.1.202   <none>        Ubuntu 22.04.3 LTS   5.15.0-1055-raspi   containerd://1.7.17-k3s1
pirack2-node3   Ready    <none>                      15h   v1.30.3+k3s1   192.168.1.203   <none>        Ubuntu 22.04.3 LTS   5.15.0-1060-raspi   containerd://1.7.17-k3s1
pirack2-node4   Ready    <none>                      15h   v1.30.3+k3s1   192.168.1.204   <none>        Ubuntu 22.04.3 LTS   5.15.0-1055-raspi   containerd://1.7.17-k3s1
pirack2-node5   Ready    <none>                      15h   v1.30.3+k3s1   192.168.1.205   <none>        Ubuntu 22.04.3 LTS   5.15.0-1055-raspi   containerd://1.7.17-k3s1

In rancher, click Import Existing -> Generic
Give it a name and click Create (I called it prod)
Apply the command, as provided by Rancher

Result

$ kubectl get all -n cattle-system
NAME                                        READY   STATUS             RESTARTS      AGE
pod/cattle-cluster-agent-678bd7b8fb-6c9fw   0/1     CrashLoopBackOff   4 (16s ago)   107s

NAME                           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/cattle-cluster-agent   ClusterIP   10.43.5.109   <none>        80/TCP,443/TCP   107s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cattle-cluster-agent   0/1     1            0           108s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/cattle-cluster-agent-678bd7b8fb   1         1         0       107s

$ kubectl describe pod cattle-cluster-agent-678bd7b8fb-6c9fw -n cattle-system
Name:         cattle-cluster-agent-678bd7b8fb-6c9fw
Namespace:    cattle-system
Priority:     0
Node:         pirack1-node3/192.168.1.103
Start Time:   Fri, 23 Aug 2024 02:47:12 +0200
Labels:       app=cattle-cluster-agent
              pod-template-hash=678bd7b8fb
Annotations:  <none>
Status:       Running
IP:           10.42.6.5
IPs:
  IP:           10.42.6.5
Controlled By:  ReplicaSet/cattle-cluster-agent-678bd7b8fb
Containers:
  cluster-register:
    Container ID:   containerd://f3ce70ea3882f2d7c74d5421785e55b8d4da2f79bb07ef6d1ff41e3f333e46f3
    Image:          rancher/rancher-agent:v2.9.0
    Image ID:       docker.io/rancher/rancher-agent@sha256:ec1299f956c811c94242cad47cde4fc5c6213effb8cf9411b9053848ef55e409
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 23 Aug 2024 02:50:18 +0200
      Finished:     Fri, 23 Aug 2024 02:50:18 +0200
    Ready:          False
    Restart Count:  5
    Environment:
      CATTLE_IS_RKE:             false
      CATTLE_SERVER:             https://<REDACTED DOMAIN>
      CATTLE_CA_CHECKSUM:
      CATTLE_CLUSTER:            true
      CATTLE_K8S_MANAGED:        true
      CATTLE_CLUSTER_REGISTRY:
      CATTLE_SERVER_VERSION:     v2.9.0
      CATTLE_INSTALL_UUID:       884f1cad-164d-4e49-85ca-ab80cd5fea48
      CATTLE_INGRESS_IP_DOMAIN:  sslip.io
      STRICT_VERIFY:             true
    Mounts:
      /cattle-credentials from cattle-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cvjh8 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  cattle-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cattle-credentials-2f2590f
    Optional:    false
  kube-api-access-cvjh8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/controlplane=true:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute for 300s
                             node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                    Message
  ----     ------     ----                   ----                    -------
  Normal   Scheduled  <unknown>                                      Successfully assigned cattle-system/cattle-cluster-agent-678bd7b8fb-6c9fw to pirack1-node3
  Normal   Pulled     2m14s (x5 over 3m44s)  kubelet, pirack1-node3  Container image "rancher/rancher-agent:v2.9.0" already present on machine
  Normal   Created    2m13s (x5 over 3m44s)  kubelet, pirack1-node3  Created container cluster-register
  Normal   Started    2m13s (x5 over 3m43s)  kubelet, pirack1-node3  Started container cluster-register
  Warning  BackOff    93s (x10 over 3m40s)   kubelet, pirack1-node3  Back-off restarting failed container cluster-register in pod cattle-cluster-agent-678bd7b8fb-6c9fw_cattle-system(f6203e00-2c33-4274-bad1-d26fd55bb452)

$ kubectl logs cattle-cluster-agent-678bd7b8fb-6c9fw -c cluster-register -n cattle-system
INFO: Environment: CATTLE_ADDRESS=10.42.6.5 CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.43.5.109:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.43.5.109:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.43.5.109 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.43.5.109:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.43.5.109 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.43.5.109 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=884f1cad-164d-4e49-85ca-ab80cd5fea48 CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-678bd7b8fb-6c9fw CATTLE_RANCHER_PROVISIONING_CAPI_VERSION= CATTLE_RANCHER_WEBHOOK_VERSION=104.0.0+up0.5.0 CATTLE_SERVER=https://<REDACTED DOMAIN> CATTLE_SERVER_VERSION=v2.9.0
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local home nameserver 10.43.0.10 options ndots:5
INFO: https://<REDACTED DOMAIN>/ping is accessible
INFO: <REDACTED DOMAIN> resolves to <REDACTED IP>
time="2024-08-23T00:53:05Z" level=info msg="Listening on /tmp/log.sock"
time="2024-08-23T00:53:05Z" level=info msg="Rancher agent version v2.9.0 is starting"
time="2024-08-23T00:53:05Z" level=error msg="unable to read CA file from /etc/kubernetes/ssl/certs/serverca: open /etc/kubernetes/ssl/certs/serverca: no such file or directory"
time="2024-08-23T00:53:05Z" level=error msg="Strict CA verification is enabled but encountered error finding root CA"

Expected Result

Rancher does it's thing and I'm able to manage the 8-node cluster from Rancher.

Screenshots

afbeelding

Additional context

To my knowledge, I have followed these same steps for previous versions of Rancher / k3s and I didn't have these problems in the past.

CC007 commented 3 weeks ago

Is this maybe related to agent-tls-mode?

CC007 commented 3 weeks ago

It was indeed related to agent-tls-mode which is set to strict by default it seems. Setting it to system-store fixed the issue.

It would be nice if the following actions were taken:

if strict is enabled, make the Rancher UI show a notice box to say that it is enabled and that either further certificate-related configuration might be required or that agent-tls-mode needs to be set to system-store (both with a link to the documentation on how to do this).
Instead of / in addition to the above error, add a log message that explains that the cattle-cluster-agent failed due to agent-tls-mode being set to strict, without having provided the server CA certificates.
Since the let's encrypt certificate is generated during the installation of the Rancher chart itself, maybe the part where the server CA cert is configured could be automated as well. I don't know if this would be during the Rancher installation or during the importing of an existing cluster (I still don't fully understand how that would even be configured).

rancher / rancher

[BUG] Unable to import existing generic cluster #46798