Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

[dyrnq]

  Normal   Pulled     34s                kubelet            Container image "docker.io/calico/cni:v3.21.5" already present on machine
  Normal   Created    34s                kubelet            Created container upgrade-ipam
  Normal   Started    34s                kubelet            Started container upgrade-ipam
  Normal   Scheduled  34s                default-scheduler  Successfully assigned kube-system/calico-node-lbmcs to n208
  Normal   Started    33s                kubelet            Started container install-cni
  Normal   Pulled     33s                kubelet            Container image "docker.io/calico/cni:v3.21.5" already present on machine
  Normal   Created    33s                kubelet            Created container install-cni
  Normal   Started    31s                kubelet            Started container flexvol-driver
  Normal   Pulled     31s                kubelet            Container image "docker.io/calico/pod2daemon-flexvol:v3.21.5" already present on machine
  Normal   Created    31s                kubelet            Created container flexvol-driver
  Normal   Pulled     30s                kubelet            Container image "docker.io/calico/node:v3.21.5" already present on machine
  Normal   Created    30s                kubelet            Created container calico-node
  Normal   Started    30s                kubelet            Started container calico-node
  Warning  Unhealthy  27s (x3 over 29s)  kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
  Warning  Unhealthy  24s                kubelet            Readiness probe failed: 2022-06-29 04:20:43.818 [INFO][438] confd/health.go 180: Number of node(s) with BGP peering established = 7
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  14s  kubelet  Readiness probe failed: 2022-06-29 04:20:53.855 [INFO][933] confd/health.go 180: Number of node(s) with BGP peering established = 7
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  4s  kubelet  Readiness probe failed: 2022-06-29 04:21:03.845 [INFO][1415] confd/health.go 180: Number of node(s) with BGP peering established = 7
calico/node is not ready: felix is not ready: readiness probe reporting 503

[dyrnq]

2022-06-29 04:30:28.883 [INFO][9111] felix/ipsets.go 312: Retrying after an ipsets update failure... family="inet"
2022-06-29 04:30:28.884 [ERROR][9111] felix/ipsets.go 574: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 04:30:28.884 [WARNING][9111] felix/ipsets.go 322: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 04:30:29.398 [ERROR][9111] felix/ipsets.go 974: Failed to read IP sets error=exit status 1 family="inet"
2022-06-29 04:30:29.398 [PANIC][9111] felix/ipsets.go 355: Failed to update IP sets after multiple retries. family="inet"
panic: (*logrus.Entry) 0xc000970230

goroutine 255 [running]:
github.com/sirupsen/logrus.Entry.log({0xc0001aa180, 0xc0001bb260, {0x0, 0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0}, 0x0, ...)
    /go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:128 +0x56c
github.com/sirupsen/logrus.(*Entry).Panic(0xc000999310, {0xc0009a9b80, 0x1, 0x1})
    /go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:173 +0xfb
github.com/projectcalico/calico/felix/ipsets.(*IPSets).ApplyUpdates(0xc000a8b400)
    /go/src/github.com/projectcalico/calico/felix/ipsets/ipsets.go:355 +0x737
github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply.func1({0x324bff0, 0xc000a8b400})
    /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:1813 +0x3d
created by github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply
    /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:1812 +0xd88

[dyrnq]

ref1 https://github.com/projectcalico/calico/issues/5717 ref2 https://github.com/rancher/rancher/issues/38017

# inxi --basic
System:
  Host: n208 Kernel: 5.15.0-39-generic x86_64 bits: 64 Console: pty pts/0
    Distro: Ubuntu 22.04 LTS (Jammy Jellyfish)
Machine:
  Type: Kvm System: QEMU product: Standard PC (Q35 + ICH9, 2009) v: pc-q35-5.2 serial: N/A
  Mobo: N/A model: N/A serial: N/A BIOS: SeaBIOS v: 1.14.0-2 date: 04/01/2014
CPU:
  Info: 4x 1-core Intel Core (Skylake IBRS) [SMP] speed (MHz): avg: 3600
Graphics:
  Device-1: driver: N/A
  Display: server: No display server data found. Headless machine? tty: 274x72
  Message: GL data unavailable in console for root.
Network:
  Device-1: Red Hat Virtio network driver: virtio-pci
Drives:
  Local Storage: total: 415 GiB used: 7 GiB (1.7%)
Info:
  Processes: 230 Uptime: 1h 58m Memory: 3.83 GiB used: 722.1 MiB (18.4%) Init: systemd
  runlevel: 5 Shell: Bash inxi: 3.3.13

ipset version
ipset v7.15, protocol version: 7

[dyrnq]

kubectl patch ds calico-node -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"calico-node","image":"docker.io/calico/node:v3.22.3"}]}}}}'