BUG Maybe: After start netbird, all Docker containers can't connect network

[johnson7788]

Describe the problem When I start netbird, No matter from netbird command or netbird docker, netbird works very well, But it leads my other docker container can't connect network. i suspect netbird conflict with docker deamon network.

Test on Linux: Ubuntu16.04 and CentOS7.

To Reproduce

Steps to reproduce the behavior:

1. install docker services
2. start netbird(docker or command)
3. Test other docker, like below
4. sudo docker run --rm --net=host -it busybox ping 123.150.76.218
5. sudo docker run --rm busybox ping so.com
6. both are failed
7. stop netbird, try again step4 and step5, success network now.

Expected behavior

A clear and concise description of what you expected to happen.

Are you using NetBird Cloud?

Please specify whether you use NetBird Cloud or self-host NetBird's control plane.

NetBird version

netbird version 0.23.3 and 0.27.7

NetBird status -d output:

If applicable, add the `netbird status -d' command output.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

[enyachoke]

@johnson7788 I also experienced this issue on CentOS 7, where the docker containers on the host could not connect to the Internet after installing Netbird.

i suspect netbird conflict with docker deamon network.

@johnson7788 have you found a way to run Netbird on a different IP range

[johnson7788]

@enyachoke My docker network is 172.17.0.0/16, and netbird is 100.124.0.0/16, I think they are already in different IP segment.

[johnson7788]

I check docker network, iptables, and routes

Start Bird, the route changed default via 192.168.50.1 dev eno1 proto static metric 100 100.124.0.0/16 dev wt0 proto kernel scope link src 100.124.182.112 169.254.0.0/16 dev eno1 scope link metric 1000 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100

IPtables: Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination

Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere localhost tcp dpt:15672 ACCEPT tcp -- anywhere localhost tcp dpt:amqp

Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references) target prot opt source destination DROP all -- anywhere anywhere RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere

Stop Bird, The route default via 192.168.50.1 dev eno1 proto static metric 100 169.254.0.0/16 dev eno1 scope link metric 1000 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100

Iptables Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination

Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:15672 ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:amqp

Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references) target prot opt source destination DROP all -- anywhere anywhere RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere

[mlsmaycon]

@enyachoke @johnson7788, can you confirm whether you have any DNS or routes enabled in NetBird's dashboard?

Also can you try forcing the use of iptables instead of nftables with the following commands:

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

[thorleifjacobsen]

@enyachoke @johnson7788 the code @mlsmaycon wrote seems to be not working for me, I changed it to this to make it use iptables instead:

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

Ref issue #2030 - seems to be a common issue on C7 or machines using nftables? I do not see this table in ubuntu on nft or iptables.

[mlsmaycon]

That's correct, I've updated the command.

The problem still happening with this flag enabled?

[thorleifjacobsen]

My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the following flag:

-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233

This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again:

chain netbird-rt-nat {
    type nat hook postrouting priority 99; policy accept;
}

I can't speak for others, but this fixed the issue for me. My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules.

My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time?

Additionally, why does this issue not occur on Ubuntu?

[thorleifjacobsen]

Another addition, in Ubuntu this is the same chain:

chain netbird-rt-nat {
        type nat hook postrouting priority srcnat - 1; policy accept;
}

Maybe CentOS NFTable Chains needs patching?

[johnson7788]

@mlsmaycon I tried it ,but it not works, see below: johnson@wacserver7:~$ cat /etc/sysconfig/netbird NB_SKIP_NFTABLES_CHECK=true johnson@wacserver7:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 371bce5deb08 netbirdio/netbird:latest "/usr/local/bin/netb…" 2 minutes ago Up 2 minutes netbirdapp 3fe3273373b6 rabbitmq:management "docker-entrypoint.s…" 3 years ago Up 3 months 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp rabbit

Docker started, but Peers ip is offline on https://app.netbird.io/peers.

cat /etc/release DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

After delete /etc/sysconfig/netbird sudo rm /etc/sysconfig/netbird sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -v netbird-client:/etc/netbird netbirdio/netbird

Peers ip is online now, i think we can't turn off NB_SKIP_NFTABLES_CHECK. but when i turn off NB_SKIP_NFTABLES_CHECK, docker network is work, like below docker run --rm --net=host -it busybox ping 123.150.76.218 PING 123.150.76.218 (123.150.76.218): 56 data bytes 64 bytes from 123.150.76.218: seq=0 ttl=52 time=10.719 ms 64 bytes from 123.150.76.218: seq=1 ttl=52 time=9.302 ms

[mlsmaycon]

hello @johnson7788 for the container, please add the environment variable as part of its running config. e.g.:

sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -e NB_SKIP_NFTABLES_CHECK=true -v netbird-client:/etc/netbird netbirdio/netbird

[mlsmaycon]

the release 0.27.8 contains the fix. Let us know if there is any other issue

[johnson7788]

@mlsmaycon Great jobs!, thank you very much, it works for me, docker network is ok now. and netbird network also works. You are so amazing

sudo docker run --rm busybox ping so.com PING so.com (180.163.237.15): 56 data bytes 64 bytes from 180.163.237.15: seq=0 ttl=51 time=29.483 ms

[johnson7788]

the release 0.27.8 contains the fix. Let us know if there is any other issue

Yes, I install on an Centos server with latest 0.27.9, Both docker network and netbird network are working. That's great!!!

[johnson7788]

My issue all be fixed. thank you everyone! @mlsmaycon @thorleifjacobsen @enyachoke