kubespan peers out of sync with nftables chains

[alexandrem]

Bug Report

Description

I've noticed an intermittent issue lately where the kubespan connectivity between my cluster nodes go away.

There seems to be a reconcilation issue between the peers that are configured in wireguard and the nftable chain entries.

As a workaround, I found that to force the reconciliation I can do a "ip link del kubespan" and everything gets back again.

Logs

wireguard entries:

interface: kubespan
  public key: UwBdfpgPLH+cNlk4L6nUhemQPXtH+m5OWQ/qgis8vwE=
  private key: (hidden)
  listening port: 51820
  fwmark: 0x20

peer: jMKJ3o8eq9Y06uDYfwyvs9p1dc/UwubTX58YHpRVLS8=
  preshared key: (hidden)
  endpoint: 20.116.223.106:51820
  allowed ips: 100.64.0.12/32, fd6c:2e1f:b2c3:9102:222:48ff:feb1:f3a/128
  latest handshake: 6 minutes, 36 seconds ago
  transfer: 878.11 KiB received, 2.01 MiB sent
  persistent keepalive: every 25 seconds

peer: S8MlhnBU08F7S9r+FSJpILv05ps1WZ1S8Jzoj7pE1AQ=
  preshared key: (hidden)
  endpoint: 100.64.0.18:51820
  allowed ips: 100.64.0.18/32, fd6c:2e1f:b2c3:9102:6245:bdff:fe61:b38/128
  latest handshake: 6 minutes, 47 seconds ago
  transfer: 10.48 MiB received, 2.91 MiB sent
  persistent keepalive: every 25 seconds

peer: +9QajiFWO+c9nQvJJgOnmz6VFPp/o/8FtQAjrAesUBo=
  preshared key: (hidden)
  endpoint: 20.116.223.105:51820
  allowed ips: 100.64.0.14/32, fd6c:2e1f:b2c3:9102:6245:bdff:fe5b:b5f0/128
  latest handshake: 7 minutes, 11 seconds ago
  transfer: 6.20 MiB received, 14.86 MiB sent
  persistent keepalive: every 25 seconds

Out of date nftables:

bash-5.1# nft list chain inet talos kubespan_prerouting
table inet talos {
        chain kubespan_prerouting {
                type filter hook prerouting priority filter; policy accept;
                meta mark & 0x00000060 == 0x00000020 accept
        }
}

bash-5.1# nft list chain inet talos kubespan_outgoing
table inet talos {
        chain kubespan_outgoing {
                type route hook output priority filter; policy accept;
                meta mark & 0x00000060 == 0x00000020 accept
                oifname "lo" accept
        }
}

After deleting the kubespan link and having it recreated:

bash-5.1# nft list chain inet talos kubespan_prerouting
table inet talos {
        chain kubespan_prerouting {
                type filter hook prerouting priority filter; policy accept;
                meta mark & 0x00000060 == 0x00000020 accept
                ip daddr { 100.64.0.12, 100.64.0.14, 100.64.0.18 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept
                ip6 daddr { fd6c:2e1f:b2c3:9102:222:48ff:feb1:f3a, fd6c:2e1f:b2c3:9102:6245:bdff:fe5b:b5f0, fd6c:2e1f:b2c3:9102:6245:bdff:fe61:b38 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept
        }
}

bash-5.1# nft list chain inet talos kubespan_outgoing
table inet talos {
        chain kubespan_outgoing {
                type route hook output priority filter; policy accept;
                meta mark & 0x00000060 == 0x00000020 accept
                oifname "lo" accept
                ip daddr { 100.64.0.12, 100.64.0.14 } tcp flags syn / syn,rst tcp option maxseg size > 1380 tcp option maxseg size set 1380
                ip6 daddr { fd6c:2e1f:b2c3:9102:222:48ff:feb1:f3a, fd6c:2e1f:b2c3:9102:6245:bdff:fe5b:b5f0 } tcp flags syn / syn,rst tcp option maxseg size > 1360 tcp option maxseg size set 1360
                ip daddr { 100.64.0.12, 100.64.0.14 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept
                ip6 daddr { fd6c:2e1f:b2c3:9102:222:48ff:feb1:f3a, fd6c:2e1f:b2c3:9102:6245:bdff:fe5b:b5f0 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept
        }
}

This occurs in both the v1.7x and v1.8.0 releases.

Environment

• Talos version: [v1.7.5, v1.8.0]
• Kubernetes version: [v1.29.9]
• Platform: bare metal, azure

[smira]

This feels similar to https://github.com/siderolabs/talos/pull/8538, but you're running a version with the fix.

Can you please submit a full talosctl support for the node when the issue happens, it would help to understand what is wrong.