NETBIRD Disrupts NAT on CentOS 7

thorleifjacobsen commented 4 months ago

Description When enabling NETBIRD on CentOS 7, NAT (Network Address Translation) stops functioning properly on two different networks (10.0.4.x and 10.0.2.x). Disabling NETBIRD restores NAT functionality. The interfaces 10.0.4.x and 10.0.2.x are not related to NETBIRD.

Current IPTables:

*nat
:PREROUTING ACCEPT [6415:852871]
:INPUT ACCEPT [6228:840858]
:OUTPUT ACCEPT [1942:137692]
:POSTROUTING ACCEPT [288:22279]
-A PREROUTING -p udp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024,1900,5353 -j DNAT --to-destination 10.0.4.236
-A PREROUTING -p tcp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024 -j DNAT --to-destination 10.0.4.236
-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233
COMMIT
# Completed on Wed May 22 13:56:11 2024
# Generated by iptables-save v1.4.21 on Wed May 22 13:56:11 2024
*filter
:INPUT ACCEPT [98280:30756187]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [51952:10249852]
-A INPUT ! -i lo -p tcp -m tcp --dport 3306 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i oc__+ -p tcp -m multiport --dports 22 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i enp0s8 -p tcp -m multiport --dports 8082 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i enp0s9 -p tcp -m multiport --dports 8080,8083 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i enp0s8 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -d 10.0.4.236/32 -p udp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024,1900,5353 -j ACCEPT
-A FORWARD -d 10.0.4.236/32 -p tcp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024 -j ACCEPT

Steps to Reproduce Set up two network interfaces with IP ranges 10.0.4.x and 10.0.2.x. Enable NAT on these interfaces. Install and enable NETBIRD. Observe that NAT stops working on both interfaces. Disable NETBIRD. Observe that NAT functionality is restored. Expected Behavior Enabling NETBIRD should not affect NAT functionality on unrelated network interfaces.

Actual Behavior Enabling NETBIRD disrupts NAT on the interfaces 10.0.4.x and 10.0.2.x. Disabling NETBIRD restores NAT functionality.

Troubleshooting Steps Taken

Verified that iptables rules remain unchanged before and after enabling NETBIRD.
Confirmed that firewalld is not installed.
Compared routing tables using ip route show before and after enabling NETBIRD (100 and 101 added ?)
Inspected network interfaces with ip addr show before and after enabling NETBIRD (no changes found other than wt0
Checked system logs (/var/log/messages and journalctl -xe) for relevant messages (no relevant entries found).
Reviewed NETBIRD configuration files for any settings related to network interfaces or routes (nothing unusual found).
Ensured NetworkManager is not making changes when NETBIRD is enabled.
Verified no other services or scripts are triggered by NETBIRD that could cause network changes.

System Information OS: CentOS 7

Additional Information

No changes in MTU settings observed.
DNS settings is irrelevant
No errors or warnings in NETBIRD logs.
Network statistics show no anomalies before and after enabling NETBIRD.

Any assistance or tips to resolve this issue would be greatly appreciated.

pascal-fischer commented 4 months ago

Hi @thorleifjacobsen,

this might be similar to https://github.com/netbirdio/netbird/issues/2015 Could you try the same solution and run

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

thorleifjacobsen commented 4 months ago

Hello! Thanks for the pointer, we did not think of checking the nft tables.

Indeed we found a problem, there is a rule in nft list tables ip netbird a chain called netbird-rt-nat by the look of it seems to prioritze over anything to accept all traffic. Which in part stops any special iptables natting.

table ip netbird {
        chain netbird-rt-fwd {
        }

       chain netbird-rt-nat {
                type nat hook postrouting priority 99; policy accept;
        }

        chain netbird-acl-input-rules {
                ip saddr & 0.0.0.0 == 0.0.0.0 accept
        }

        chain netbird-acl-output-rules {
                ip daddr & 0.0.0.0 == 0.0.0.0 accept
        }

        chain netbird-acl-input-filter {
                type filter hook input priority 0; policy accept;
                iifname "wt0" ip saddr 100.66.0.0/16 ip daddr != 100.66.0.0/16 accept
                iifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.0.0/16 accept
                iifname "wt0" ip saddr 100.66.0.0/16 ip daddr 100.66.0.0/16 jump netbird-acl-input-rules
                iifname "wt0" drop
        }

        chain netbird-acl-output-filter {
                type filter hook output priority 0; policy accept;
                oifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.0.0/16 accept
                oifname "wt0" ip saddr 100.66.0.0/16 ip daddr != 100.66.0.0/16 accept
                oifname "wt0" ip saddr 100.66.0.0/16 ip daddr 100.66.0.0/16 jump netbird-acl-output-rules
                oifname "wt0" drop
        }

        chain netbird-acl-forward-filter {
                type filter hook forward priority 0; policy accept;
                iifname "wt0" jump netbird-rt-fwd
                oifname "wt0" jump netbird-rt-fwd
                iifname "wt0" mark 0x000007e4 accept
                oifname "wt0" mark 0x000007e4 accept
                iifname "wt0" jump netbird-acl-input-rules
                iifname "wt0" drop
        }

        chain netbird-acl-prerouting-filter {
                type filter hook prerouting priority -150; policy accept;
                iifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.121.137 mark set 0x000007e4
        }
}

Removing netbird-rt-nat made everything work again.

Your solution seems to work but I had to add =true as the code seems to be depended on this: os.Getenv(SKIP_NFTABLES_ENV) != "true"

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

But is this a bug that nftables is breaking any further specialised natting? Having the rule with priority 99 which just accepts with no special spesifications is that OK? Maybe we have a real issue at hand here?

thorleifjacobsen commented 4 months ago

My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the script as shown above. This was the NAT rule which was previousl ignored:

-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233

This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again:

# From CentOS NFTables
chain netbird-rt-nat {
    type nat hook postrouting priority 99; policy accept;
}

My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules.

My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time?

Additionally, why does this issue not occur on Ubuntu? After checking Ubunti I saw they have a lower priority on their chain in nftables?

Is this a bug which should be fixed for CentOS?

# From Ubuntu NFTables
chain netbird-rt-nat {
        type nat hook postrouting priority srcnat - 1; policy accept;
}

mlsmaycon commented 4 months ago

Thanks for the update, @thorleifjacobsen; yes. It is a bug; with iptables and nftables competing for precedence, this rule should not be in place in case there were no rules to be added to it.

It seems like the best approach might be to detect an old kernel or iptables versions that don't support nftables and set the client to use iptables instead.

thorleifjacobsen commented 4 months ago

Keep in mind, this rule is available both on Ubuntu and CentOS (but it seems to be srcnat - 1 in priority vs 99 in CentOS).

When changing to iptables it adds some similar rules to iptables:

*nat
:PREROUTING ACCEPT [12:1383]
:INPUT ACCEPT [12:1383]
:OUTPUT ACCEPT [14:1225]
:POSTROUTING ACCEPT [14:1225]
:NETBIRD-RT-NAT - [0:0]
-A PREROUTING -p udp -m mac --mac-source 96:4F:1D:1F:69:CA -m multiport ! --dports 0:1024,1900,5353 -j DNAT --to-destination 10.0.4.236
-A PREROUTING -p tcp -m mac --mac-source 96:4F:1D:1F:69:CA -m multiport ! --dports 0:1024,8008,8443 -j DNAT --to-destination 10.0.4.236
-A POSTROUTING -j NETBIRD-RT-NAT
-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233
-A NETBIRD-RT-NAT -j RETURN

mlsmaycon commented 4 months ago

Thanks, @thorleifjacobsen; for the next release, we will verify if iptables and nftables can work together and fall back to iptables by default.

It is better to have a single firewall manager dealing with packets.

mlsmaycon commented 4 months ago

the release 0.27.8 contains the fix. Let us know if there is any other issue

thorleifjacobsen commented 4 months ago

0.27.9 installed and it works as expected. Thanks :)

netbirdio / netbird

NETBIRD Disrupts NAT on CentOS 7 #2030