netbirdio / netbird

Connect your devices into a single secure private WireGuard®-based mesh network with SSO/MFA and simple access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
9.87k stars 428 forks source link

Docker container or K8s pod do not send request to host, but correct resolve host to ip. #2063

Open gawsoftpl opened 1 month ago

gawsoftpl commented 1 month ago

Describe the problem

Netbird in docker container or in kubernetes pod for self hosted netbird. Dns resolve correct ip (nslookup works correct) but curl, ping, wget return error: bad address For normal ubuntu instance without docker or kubernetes evereythings works correct

How I run docker

docker run --rm -d 
 --cap-add=NET_ADMIN  
 -e NB_SETUP_KEY=<REDACTED>
 -v netbird-client:/etc/netbird  
 -e NB_MANAGEMENT_URL=<REDACTED>
 netbirdio/netbird:latest

How I run k8s pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: netbird
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: netbird
  template:
    metadata:
      labels:
        app: netbird
    spec:
      containers:
        - name: netbird
          image: netbirdio/netbird:latest
          env:
            - name: NB_MANAGEMENT_URL
              value: "<REDACTED>"
            - name: NB_SETUP_KEY
              value: "<REDACTED>" # replace with your setup key
          securityContext:
            capabilities:
              add:
                - NET_ADMIN

Example commands executed in docker container or in k8s pods. Nslookup return correct ip 100.87.97.114

nslookup netbird-client-1-2.netbird.selfhosted
Server:     100.87.150.177
Address:    100.87.150.177:53

** server can't find netbird-client-1-2.netbird.selfhosted: NXDOMAIN

Non-authoritative answer:
Name:   netbird-client-1-2.netbird.selfhosted
Address: 100.87.97.114

Ping with ip works correct

/ # ping 100.87.97.114
PING 100.87.97.114 (100.87.97.114): 56 data bytes
64 bytes from 100.87.97.114: seq=0 ttl=64 time=1.503 ms
64 bytes from 100.87.97.114: seq=1 ttl=64 time=1.032 ms

Wget with ip works correct

wget 100.87.97.114:5555
Connecting to 100.87.97.114:5555 (100.87.97.114:5555)
saving to 'index.html'
index.html           100% |******************************************************|     5  0:00:00 ETA
'index.html' saved

Error here

But when I execute client http request or ping for host I received dns resolve error:

ping netbird-client-1-2.netbird.selfhosted
ping: bad address 'netbird-client-1-2.netbird.selfhosted'
wget netbird-client-1-2.netbird.selfhosted
wget: bad address 'netbird-client-1-2.netbird.selfhosted'
wget netbird-client-1-2:5555
wget: bad address 'netbird-client-1-2:5555'

Expected behavior

Ping, curl or wget should send request

Are you using NetBird Cloud?

Netbird self hosted

NetBird version

0.27.10

NetBird status -d output:

I run netbird in container, I cant execute status output

bcmmbaga commented 1 month ago

Hello @gawsoftpl, can you try testing again running client with these additional capabilities: --cap-add=SYS_ADMIN and --cap-add=SYS_RESOURCE ?

gawsoftpl commented 1 month ago

Hello @gawsoftpl, can you try testing again running client with these additional capabilities: --cap-add=SYS_ADMIN and --cap-add=SYS_RESOURCE ?

This not help in docker and k8s same issue.

I past resolv conf

cat /etc/resolv.conf
# Generated by NetBird
# If needed you can restore the original file by copying back /etc/resolv.conf.original.netbird

options timeout:4 attempts:1
search netbird.selfhosted
nameserver 100.87.0.130
nameserver 100.87.231.247
nameserver 185.12.64.1
nameserver 185.12.64.2
cat /etc/nsswitch.conf 
# musl itself does not support NSS, however some third-party DNS
# implementations use the nsswitch.conf file to determine what
# policy to follow.
# Editing this file is not recommended.
hosts: files dns
cat /etc/resolv.conf.original.netbird
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 100.87.231.247
nameserver 185.12.64.1
nameserver 185.12.64.2
search netbird.selfhosted

# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: []

And ifconfig from docker container

eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02  
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:948 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1107 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:135010 (131.8 KiB)  TX bytes:137313 (134.0 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:39 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3898 (3.8 KiB)  TX bytes:3898 (3.8 KiB)

wt0       Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:100.87.0.130  P-t-P:100.87.0.130  Mask:255.255.0.0
          UP POINTOPOINT RUNNING NOARP  MTU:1280  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:5 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1364 (1.3 KiB)  TX bytes:1928 (1.8 KiB)
bcmmbaga commented 1 month ago

@gawsoftpl you can enable the debug logs with -e NB_LOG_LEVEL=debug and share the client logs

gawsoftpl commented 1 month ago

logs from docker container: logs.txt