calico-node init failed with pod error `Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused`

[chenm0520]

I am trying to build a lastest k8s cluster in production environment,following the Official documentation Install using Helm to install calico without values.yaml. After installation，the calico-node pod can't be healthly

Your Environment

• Calico version v3.27.3
• Orchestrator version (e.g. kubernetes, mesos, rkt): k8s 1.30.0
• Operating System and version: Alibaba Cloud Linux 3.2104 $kubectl describe pod calico-node-ttkpm -n calico-system `Name: calico-node-wpght Namespace: calico-system Priority: 2000001000 Priority Class Name: system-node-critical Service Account: calico-node Node: node/172.29.225.217 Start Time: Tue, 23 Apr 2024 00:59:53 +0800 Labels: app.kubernetes.io/name=calico-node controller-revision-hash=759f6bdfdd k8s-app=calico-node pod-template-generation=1 Annotations: hash.operator.tigera.io/cni-config: 7c3d4f43aed6c9f83376215a5e9a368bef0902a0 hash.operator.tigera.io/system: fdde45054a8ae4f629960ce37570929502e59449 tigera-operator.hash.operator.tigera.io/tigera-ca-private: 6a3b26e24035401f2a37e6910a29c987244ab792 Status: Running IP: 172.29.225.217 IPs: IP: 172.29.225.217 Controlled By: DaemonSet/calico-node Init Containers: flexvol-driver: Container ID: containerd://ec434aa491dcb113c9cc1efa70f950e7e7d58c4bb0435d2e400bbd597a4198ea Image: docker.io/calico/pod2daemon-flexvol:v3.27.3 Image ID: docker.io/calico/pod2daemon-flexvol@sha256:7c559ca2c1f476caee87c8db6140d017e8bc2f1f1b0b57103c33dba172c09097 Port: Host Port: SeccompProfile: RuntimeDefault State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 23 Apr 2024 00:59:54 +0800 Finished: Tue, 23 Apr 2024 00:59:54 +0800 Ready: True Restart Count: 0 Environment: Mounts: /host/driver from flexvol-driver-host (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fv4c7 (ro) install-cni: Container ID: containerd://ce4ad7b7869b4f9fc6c5be4d69b285251c37210f9b820c01aa8e33a7ac20f922 Image: docker.io/calico/cni:v3.27.3 Image ID: docker.io/calico/cni@sha256:1f2c6a13d436b2ae056edd46552d23279d3aaf5d79152fb88cd959b634acfd6f Port: Host Port: SeccompProfile: RuntimeDefault Command: /opt/cni/bin/install State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 23 Apr 2024 00:59:55 +0800 Finished: Tue, 23 Apr 2024 00:59:57 +0800 Ready: True Restart Count: 0 Environment: CNI_CONF_NAME: 10-calico.conflist SLEEP: false CNI_NET_DIR: /etc/cni/net.d CNI_NETWORK_CONFIG: <set to the key 'config' of config map 'cni-config'> Optional: false KUBERNETES_SERVICE_HOST: 10.96.0.1 KUBERNETES_SERVICE_PORT: 443 Mounts: /host/etc/cni/net.d from cni-net-dir (rw) /host/opt/cni/bin from cni-bin-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fv4c7 (ro) Containers: calico-node: Container ID: containerd://74c570c3236a302657261a903401a2eccde6f6481f6f7a2ec0654ce1b18195f3 Image: docker.io/calico/node:v3.27.3 Image ID: docker.io/calico/node@sha256:78dadbe45a1a76f685d5fa777fdec807a86368b8e748cb27dad14723e233803c Port: Host Port: SeccompProfile: RuntimeDefault State: Running Started: Tue, 23 Apr 2024 00:59:58 +0800 Ready: True Restart Count: 0 Liveness: http-get http://localhost:9099/liveness delay=0s timeout=10s period=60s #success=1 #failure=3 Readiness: exec [/bin/calico-node -bird-ready -felix-ready] delay=0s timeout=10s period=30s #success=1 #failure=3 Environment: DATASTORE_TYPE: kubernetes WAIT_FOR_DATASTORE: true CLUSTER_TYPE: k8s,operator,bgp CALICO_DISABLE_FILE_LOGGING: false FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT FELIX_HEALTHENABLED: true FELIX_HEALTHPORT: 9099 NODENAME: (v1:spec.nodeName) NAMESPACE: calico-system (v1:metadata.namespace) FELIX_TYPHAK8SNAMESPACE: calico-system FELIX_TYPHAK8SSERVICENAME: calico-typha FELIX_TYPHACAFILE: /etc/pki/tls/certs/tigera-ca-bundle.crt FELIX_TYPHACERTFILE: /node-certs/tls.crt FELIX_TYPHAKEYFILE: /node-certs/tls.key FIPS_MODE_ENABLED: false FELIX_TYPHACN: typha-server CALICO_MANAGE_CNI: true CALICO_IPV4POOL_CIDR: 10.95.0.0/16 CALICO_IPV4POOL_IPIP: Always CALICO_IPV4POOL_BLOCK_SIZE: 26 CALICO_IPV4POOL_NODE_SELECTOR: all() CALICO_IPV4POOL_DISABLE_BGP_EXPORT: false CALICO_NETWORKING_BACKEND: bird IP: autodetect IP_AUTODETECTION_METHOD: first-found IP6: none FELIX_IPV6SUPPORT: false KUBERNETES_SERVICE_HOST: 10.96.0.1 KUBERNETES_SERVICE_PORT: 443 Mounts: /etc/pki/tls/cert.pem from tigera-ca-bundle (ro,path="ca-bundle.crt") /etc/pki/tls/certs from tigera-ca-bundle (ro) /host/etc/cni/net.d from cni-net-dir (rw) /lib/modules from lib-modules (ro) /node-certs from node-certs (ro) /run/xtables.lock from xtables-lock (rw) /var/lib/calico from var-lib-calico (rw) /var/log/calico/cni from cni-log-dir (rw) /var/run/calico from var-run-calico (rw) /var/run/nodeagent from policysync (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fv4c7 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lib-modules: Type: HostPath (bare host directory volume) Path: /lib/modules HostPathType:
  xtables-lock: Type: HostPath (bare host directory volume) Path: /run/xtables.lock HostPathType: FileOrCreate policysync: Type: HostPath (bare host directory volume) Path: /var/run/nodeagent HostPathType: DirectoryOrCreate tigera-ca-bundle: Type: ConfigMap (a volume populated by a ConfigMap) Name: tigera-ca-bundle Optional: false node-certs: Type: Secret (a volume populated by a Secret) SecretName: node-certs Optional: false var-run-calico: Type: HostPath (bare host directory volume) Path: /var/run/calico HostPathType:
  var-lib-calico: Type: HostPath (bare host directory volume) Path: /var/lib/calico HostPathType:
  cni-bin-dir: Type: HostPath (bare host directory volume) Path: /opt/cni/bin HostPathType:
  cni-net-dir: Type: HostPath (bare host directory volume) Path: /etc/cni/net.d HostPathType:
  cni-log-dir: Type: HostPath (bare host directory volume) Path: /var/log/calico/cni HostPathType:
  flexvol-driver-host: Type: HostPath (bare host directory volume) Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds HostPathType: DirectoryOrCreate kube-api-access-fv4c7: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: kubernetes.io/os=linux Tolerations: :NoSchedule op=Exists :NoExecute op=Exists CriticalAddonsOnly op=Exists node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message

  ---

  Normal Scheduled 5m40s default-scheduler Successfully assigned calico-system/calico-node-wpght to node Normal Pulled 5m39s kubelet Container image "docker.io/calico/pod2daemon-flexvol:v3.27.3" already present on machine Normal Created 5m39s kubelet Created container flexvol-driver Normal Started 5m39s kubelet Started container flexvol-driver Normal Pulled 5m38s kubelet Container image "docker.io/calico/cni:v3.27.3" already present on machine Normal Created 5m38s kubelet Created container install-cni Normal Started 5m38s kubelet Started container install-cni Normal Pulled 5m36s kubelet Container image "docker.io/calico/node:v3.27.3" already present on machine Normal Created 5m35s kubelet Created container calico-node Normal Started 5m35s kubelet Started container calico-node Warning Unhealthy 5m33s (x2 over 5m34s) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused`

cat calico-node's container log, some error log: ` 2024-04-23T01:00:01.370346526+08:00 2024-04-22 17:00:01.367 [ERROR][75] confd/resource.go 318: Error from checkcmd "bird -p -c /etc/calico/confd/config/.bird.cfg2832451251": "bird: /etc/calico/confd/config/.bird.cfg2832451251:6:1 Unable to open included file /etc/calico/confd/config/bird_aggr.cfg: No such file or directory\n"

2024-04-23T01:00:01.370348574+08:00 2024-04-22 17:00:01.367 [INFO][75] confd/resource.go 252: Check failed, but file does not yet exist - create anyway `

It was the last first that I built k8s cluster and I have tried many times,any useful troubleshooting ideas would be helpful, thanks!

[chencjcj]

Hello, I also encountered this problem, have you solved it?

[chenm0520]

Hello, I also encountered this problem, have you solved it? well,What should I say? I overloaded the system and run the installation script ，then everything is ok,the biggest difference between this time and the previous one maybe the addition of kernel model nf_conntrack(case official document shows nf_conntrack_ipv4,it can't be recognized by the my system's kernel,I removed it),hope this can help you

[csthink]

I also faced this problem, any work around?