netbird 0.32.0 breaks K3s 1.32.2+k3s1 with flannel due to iptables conflicts

[christian-schlichtherle]

Describe the problem

We're operating an IoT project where some K3s nodes are placed at customer premises. So we are installing Netbird 0.32.0 on each node first and then install K3s v1.32.2+k3s1 using flannel next. When installing k3s, we are providing flannel-iface=wt0 to tell it to use the Netbird interface for node-to-node communication.

This works great to some extent but there is a problem: When the Netbird service starts, it sets up its iptables rules. Also, flannel sets up its iptables rules. However, there seem to be conflicts in those rules, resulting in communication being broken after every restart of the Netbird service, e.g. when installing an upgrade. As a workaround, I have to restart the k3s(-agent) service after every restart of the Netbird service.

Summing it up, to restart all Netbird services in the cluster, I have to do something like this:

ansible k3s_server -b -m shell --forks 1 -a 'systemctl restart netbird && sleep 3 && systemctl restart k3s'
ansible k3s_agent -b -m shell -a 'systemctl restart netbird && sleep 3 && systemctl restart k3s-agent'

As you can imagine, this is not a sustainable solution, just a hacky workaround.

Is this a known issue? What are my options? Wait for a fix or try another CNI like cilium?

To Reproduce

Steps to reproduce the behavior:

• Install Netbird on a bunch of nodes
• Install K3s on the nodes with flannel-iface=wt0
• Restart the netbird service only and watch the in-cluster communication to break, e.g. you can't kubectl logs <any-pod> anymore.

Expected behavior

Not breaking the in-cluster communication by leaving flannel's iptable rules alone.

Are you using NetBird Cloud?

Yes

NetBird version

0.32.0

NetBird status -dA output:

n/a

Do you face any (non-mobile) client issues?

Yes.

Screenshots

n/a

Additional context

See above.

[christian-schlichtherle]

BTW: This is a long-standing problem, I just had no time to report it earlier.

[lixmal]

Hi @christian-schlichtherle, can you post your iptables/nftables before and after your workaround?

iptables-save
nft list ruleset

You might need to install nftables for the nft tool to be available