[BUG] cluster creation stuck waiting for cluster agent to connect ipv6

vcharlet commented 11 months ago

Rancher Server Setup

Rancher version: v2.8.0
Local Cluster : RKE2 Helm v1.26.11+rke2r1
user role : admin

Describe the bug All nodes are IPv6 only (local and downstream).

On a fresh Rancher v2.8.0, i tried to create a custom RKE2 cluster with 1 node. The cluster is stuck on waiting for cluster agent to connect.

Additional informations The cluster is an IPv6 only cluster, so it has been configured with additional options to avoid being stuck with probes: https://github.com/rancher/rancher/issues/42411

Exemple :

cluster-cidr: "2001:cafe:42:0::/56"
service-cidr: "2001:cafe:42:1::/112"

spec:
  rkeConfig:
    networking:
      stackPreference: ipv6

Screenshots

Machine status :

The downstream cluster seems fine :

cluster-agent on downstream cluster seems fine too. There are no errors in logs. I tried to restart cluster-agent without success.

rancher.lan is the rancher hostname

i tried to open a shell on cluster-agent pod to do some tests :

dns is fine (dig rancher.lan aaaa give the ipv6, it has no ipv4)
i can connect with curl -v https://rancher.lan

full logs

INFO: Environment: CATTLE_ADDRESS= CATTLE_CA_CHECKSUM=3eb92a84040e444821b4af926c6806f5697568824c4aa5075b4e6b2f1dc2d268 CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://[2b31:3440:c43:1b::a55d]:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://[2b31:3440:c43:1b::a55d]:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=2b31:3440:c43:1b::a55d CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://[2b31:3440:c43:1b::a55d]:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=2b31:3440:c43:1b::a55d CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=2b31:3440:c43:1b::a55d CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_FEATURES=embedded-cluster-api=false,fleet=false,monitoringv1=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningv2=false,rke2=false CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=8ea8426f-7fb7-480b-a47d-f2f0cf5649fe CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-5bc9698559-lnsvx CATTLE_RANCHER_WEBHOOK_VERSION=103.0.1+up0.4.2 CATTLE_SERVER=https://rancher.lan CATTLE_SERVER_VERSION=v2.8.0
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local hosts.lan nameserver 2b31:3440:c43:1b::a options ndots:5
INFO: https://rancher.lan/ping is accessible
INFO: rancher.lan resolves to 
INFO: Value from https://rancher.lan/v3/settings/cacerts is an x509 certificate
time="2023-12-26T22:16:30Z" level=info msg="Listening on /tmp/log.sock"
time="2023-12-26T22:16:30Z" level=info msg="Rancher agent version v2.8.0 is starting"
time="2023-12-26T22:16:30Z" level=info msg="Connecting to wss://rancher.lan/v3/connect/register with token starting with p8mv4r6lcb2dqmgdn5kgnjndfn7"
time="2023-12-26T22:16:30Z" level=info msg="Connecting to proxy" url="wss://rancher.lan/v3/connect/register"
time="2023-12-26T22:16:30Z" level=info msg="Starting /v1, Kind=Service controller"
time="2023-12-26T22:16:30Z" level=info msg="Running in single server mode, will not peer connections"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD features.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD navlinks.ui.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD clusters.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD apiservices.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD clusterregistrationtokens.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD settings.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD preferences.management.cattle.io"
time="2023-12-26T22:16:30Z" level=info msg="Applying CRD features.management.cattle.io"
time="2023-12-26T22:16:31Z" level=info msg="Applying CRD clusterrepos.catalog.cattle.io"
time="2023-12-26T22:16:31Z" level=info msg="Applying CRD operations.catalog.cattle.io"
time="2023-12-26T22:16:31Z" level=info msg="Applying CRD apps.catalog.cattle.io"
time="2023-12-26T22:16:31Z" level=info msg="Starting API controllers"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Group controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Cluster controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=User controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Token controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=UserAttribute controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=GroupMember controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Feature controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=Role controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting catalog.cattle.io/v1, Kind=ClusterRepo controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Cluster controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Setting controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=Preference controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting apiregistration.k8s.io/v1, Kind=APIService controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=Namespace controller"
time="2023-12-26T22:16:31Z" level=info msg="Starting management.cattle.io/v3, Kind=APIService controller"
I1226 22:16:31.869780      56 leaderelection.go:245] attempting to acquire leader lease kube-system/cattle-controllers...
time="2023-12-26T22:16:31Z" level=info msg="Starting steve aggregation client"
time="2023-12-26T22:16:31Z" level=info msg="Listening on :443"
time="2023-12-26T22:16:31Z" level=info msg="certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1703628502,O=dynamiclistener-org: notBefore=2023-12-26 22:08:22 +0000 UTC notAfter=2024-12-25 22:16:31 +0000 UTC"
time="2023-12-26T22:16:31Z" level=warning msg="dynamiclistener [::]:443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2023-12-26T22:16:31Z" level=info msg="Active TLS secret / (ver=) (count 4): map[listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-5ef164:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d3 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=E18DDE55CE42799E5BDF0C9A2497AFC0E8267EC0]"
time="2023-12-26T22:16:31Z" level=info msg="Listening on :80"
time="2023-12-26T22:16:31Z" level=info msg="Active TLS secret cattle-system/serving-cert (ver=2473) (count 6): map[field.cattle.io/projectId:c-m-lh7qt27d:p-75hlq listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-48d210:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d0 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-6e9ea6:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4c2 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=85889003B3475CB1D94049888BF97A844CC627A9]"
time="2023-12-26T22:16:31Z" level=info msg="Listening on :444"
time="2023-12-26T22:16:31Z" level=warning msg="dynamiclistener [::]:444: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2023-12-26T22:16:31Z" level=info msg="certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1703628502,O=dynamiclistener-org: notBefore=2023-12-26 22:08:22 +0000 UTC notAfter=2024-12-25 22:16:31 +0000 UTC"
time="2023-12-26T22:16:31Z" level=info msg="Updating TLS secret for cattle-system/serving-cert (count: 7): map[field.cattle.io/projectId:c-m-lh7qt27d:p-75hlq listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-48d210:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d0 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-5ef164:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d3 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-6e9ea6:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4c2 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=784F1CE191EEE73DB7300E4097210AE379912273]"
time="2023-12-26T22:16:31Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-12-26T22:16:31Z" level=info msg="Updating TLS secret for cattle-system/serving-cert (count: 7): map[field.cattle.io/projectId:c-m-lh7qt27d:p-75hlq listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-48d210:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d0 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-5ef164:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d3 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-6e9ea6:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4c2 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=784F1CE191EEE73DB7300E4097210AE379912273]"
time="2023-12-26T22:16:31Z" level=info msg="Active TLS secret cattle-system/serving-cert (ver=5233) (count 7): map[field.cattle.io/projectId:c-m-lh7qt27d:p-75hlq listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-48d210:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d0 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-5ef164:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4d3 listener.cattle.io/cn-2b31_3440_c43_1a_bb3f_b3b0_d25b_b4-6e9ea6:2b31:3440:c43:1a:bb3f:b3b0:d25b:b4c2 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=784F1CE191EEE73DB7300E4097210AE379912273]"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for discovery.k8s.io/v1, Kind=EndpointSlice"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=PersistentVolume"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for storage.k8s.io/v1, Kind=StorageClass"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for networking.k8s.io/v1, Kind=IngressClass"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=GroupMember"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apps/v1, Kind=Deployment"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=ConfigMap"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Endpoints"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Event"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for snapshot.storage.k8s.io/v1, Kind=VolumeSnapshotClass"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=BGPFilter"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for networking.k8s.io/v1, Kind=Ingress"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=PersistentVolumeClaim"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for snapshot.storage.k8s.io/v1, Kind=VolumeSnapshot"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for certificates.k8s.io/v1, Kind=CertificateSigningRequest"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apps/v1, Kind=ReplicaSet"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for policy/v1, Kind=PodDisruptionBudget"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apps/v1, Kind=DaemonSet"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for helm.cattle.io/v1, Kind=HelmChart"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apiregistration.k8s.io/v1, Kind=APIService"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for operator.tigera.io/v1, Kind=Installation"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for k3s.cattle.io/v1, Kind=Addon"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Namespace"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Node"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=AuthConfig"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Secret"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=UserAttribute"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for helm.cattle.io/v1, Kind=HelmChartConfig"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for operator.tigera.io/v1, Kind=ImageSet"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=APIService"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for storage.k8s.io/v1, Kind=CSIStorageCapacity"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for operator.tigera.io/v1, Kind=APIServer"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Group"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=PodSecurityAdmissionConfigurationTemplate"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for node.k8s.io/v1, Kind=RuntimeClass"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Feature"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for autoscaling/v2, Kind=HorizontalPodAutoscaler"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for flowcontrol.apiserver.k8s.io/v1beta3, Kind=PriorityLevelConfiguration"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Cluster"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=ReplicationController"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for storage.k8s.io/v1, Kind=CSIDriver"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=KubeControllersConfiguration"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for operator.tigera.io/v1, Kind=TigeraStatus"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for snapshot.storage.k8s.io/v1, Kind=VolumeSnapshotContent"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for catalog.cattle.io/v1, Kind=Operation"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=User"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=ServiceAccount"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Token"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Pod"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Setting"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=LimitRange"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for events.k8s.io/v1, Kind=Event"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for catalog.cattle.io/v1, Kind=App"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apps/v1, Kind=ControllerRevision"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for batch/v1, Kind=CronJob"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for storage.k8s.io/v1, Kind=CSINode"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for ui.cattle.io/v1, Kind=NavLink"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for k3s.cattle.io/v1, Kind=ETCDSnapshotFile"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=PodTemplate"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=Service"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for apps/v1, Kind=StatefulSet"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Preference"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for /v1, Kind=ResourceQuota"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=IPReservation"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=IPPool"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=CalicoNodeStatus"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for batch/v1, Kind=Job"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration"
time="2023-12-26T22:16:32Z" level=info msg="Watching metadata for coordination.k8s.io/v1, Kind=Lease"
I1226 22:16:36.265432      56 leaderelection.go:255] successfully acquired lease kube-system/cattle-controllers
time="2023-12-26T22:16:36Z" level=info msg="Steve auth startup complete"
time="2023-12-26T22:16:36Z" level=info msg="Registering namespaceHandler for adding labels "
time="2023-12-26T22:16:36Z" level=info msg="Starting apps/v1, Kind=Deployment controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting apps/v1, Kind=ReplicaSet controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting catalog.cattle.io/v1, Kind=Operation controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting /v1, Kind=ReplicationController controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting /v1, Kind=Service controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting catalog.cattle.io/v1, Kind=App controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting apps/v1, Kind=StatefulSet controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting networking.k8s.io/v1, Kind=Ingress controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting /v1, Kind=Node controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting batch/v1, Kind=CronJob controller"
time="2023-12-26T22:16:36Z" level=info msg="Starting /v1, Kind=Pod controller"

Thanks for helping.

vcharlet commented 11 months ago

After more digging, i found a lot of errors in fleet-controller logs.

2023-12-26T23:18:31.027085701Z time="2023-12-26T23:18:31Z" level=error msg="error syncing 'fleet-default/test-12': handler import-cluster: host must be a URL or a host:port pair: \"https://2b31:3440:c00:1b::9be3/k8s/clusters/c-m-lh7qt27d\", requeuing"

It seems to be the problem.

jakefhyde commented 10 months ago

@vcharlet For future reference, ipv6 urls have to be encolsed in brackets (like this: https://[2b31:3440:c00:1b::9be3] so your rancher server-url is not a valid URL, which will prevent the cluster agents from dialing back to rancher as it cannot be resolved. Once that issue is alleviated, feel free to reach back out as I want to make sure it's working.

jakefhyde commented 9 months ago

@vcharlet Were you able to alleviate the issue?

vcharlet commented 9 months ago

@jakefhyde Sorry for the delay.

I couldn't solve the problem. I know IPv6 have to be enclosed in brackets and my server url is good, it's a domain name.

I'm trying to deploy a custom RKE2 cluster from the dashboard with the registration command.

The provisioned cluster itself seems fine, all pods are OK, all probes are OK, i can access it with kubetl, etc ... The provisioning is just stuck at the end and i can't access the cluster in the dashboard.

The only error i can find is in fleet-controller logs. Maybe this issue is related : https://github.com/rancher/rancher/issues/42722 https://github.com/rancher/rancher/blob/4cf3b4a6e94f99b8ef78bf8f254d9e62fdf400cc/cmd/agent/main.go#L367

serverURL.Host return an IPv6 with no brackets ? It's not the same error than mine, I've a problem with /k8s/clusters/**.

I've not set this IP anywhere manually and it's not my server-url. It's the IP of the Rancher service on the local cluster.

I can do more tests but i don't know where to look for.

Thanks for helping

vcharlet commented 9 months ago

https://github.com/rancher/rancher/blob/4cf3b4a6e94f99b8ef78bf8f254d9e62fdf400cc/pkg/controllers/dashboard/apiservice/setting.go#L53

https://github.com/rancher/rancher/blob/4cf3b4a6e94f99b8ef78bf8f254d9e62fdf400cc/pkg/controllers/dashboard/apiservice/setting.go#L85

I think there is a problem here. service.Spec.ClusterIP is an IPv6. The url is malformed and brackets are missing.

Could this be the problem ?

jakefhyde commented 8 months ago

@vcharlet That may very well be the issue, makes sense since fleet consumes that as part of the kubeconfig secret.

vcharlet commented 8 months ago

@jakefhyde Yes, I've done quite a few tests and I'm pretty sure that's the problem.

This has to do with the fact that the local cluster where rancher is installed is ipv6 only.

The service.Spec.ClusterIP of the rancher service is an IPv6.

It's easy to reproduce.

The error is also present in the dashboard in the "Clusters" tab for all cluster (local cluster included).

0 Nodes Ready despite the fact that the cluster works well.

jakefhyde commented 8 months ago

@olblak I'm assigning this over to the fleet team, since this is related to the kubeconfig secret that we generate for the local cluster.

thardeck commented 2 months ago

This should be fixed in the current main branch.

olblak commented 2 months ago

This should be fixed in the current main branch.

So I guess we are targeting the Rancher 2.10 Is this something we need to backport to 2.9.3/2.8.7?

@Jono-SUSE-Rancher Do you have any opinion?

Jono-SUSE-Rancher commented 2 months ago

Hi @olblak - If you want to target it for v2.10.0, we should move it into that milestone. In terms of whether or not we should backport it, I mean, I would think we definitely want to fix it in v2.9.3. I would check with Cam to see if we need it in v2.8x.

thardeck commented 2 months ago

Documentation regarding QA testing considerations and tests in general are documented in my pr at the top.

Jono-SUSE-Rancher commented 1 day ago

@kkaempf - Can we close this? I am going to close the milestone since we released on Monday.

rancher / rancher

[BUG] cluster creation stuck waiting for cluster agent to connect ipv6 #43878