search

netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
10.74k stars 484 forks source link

DNS resolution fails when running a local resolver #2157

Open MDMeridio001 opened 3 months ago

MDMeridio001 commented 3 months ago

Describe the problem

I run a local DNS resolver (unbound) on a netbird peer running Ubuntu Server 22.04.4 LTS. DNS management is disabled for the peer in question via the netbird dashboard. Everything worked fine in version 0.27.10, but after updating to 0.28.0 the peer is not able to reach the unbound server running on localhost and every DNS query results in a timeout.

To Reproduce

Steps to reproduce the behavior:

  1. Run "netbird up" on the peer
  2. Run "dig google.com @127.0.0.1" on the peer
  3. dig command results in ";; communications error to 127.0.0.1#53: timed out"
  4. Run "netbird down" on the peer
  5. DNS resolution is back to normal

Expected behavior

I expect the DNS resolution to not be impacted whether or not the peer is connected to netbird, especially because DNS management is disabled in the dashboard for the said peer.

Are you using NetBird Cloud?

No, I'm self-hosting it.

NetBird version

0.28.0

mrota83 commented 3 months ago

same thing here worked fine before 0.28.0

nikolicjakov commented 3 months ago

I am having same problem running new 0.28.0 version. And i am also self hosting.

WARN [error: read udp 192.168.25.40:56297->192.168.40.168:53: i/o timeout, upstream: 192.168.40.168:53] client/internal/dns/upstream.go:101: got an error while connecting to upstream

DNS_PROBE_FINISHED_NXDOMAIN

I have 2 DNS servers running in private network and i have created network route to this subnet so that anyone who is connected to vpn can access these DNS servers (one of them 192.168.40.168). But for some reason after 0.28.0 update i am only experiencing issues with internal domain resolutions and i am seeing file getting created every 10 seconds.

And if i am trying to open internal domain client connection breaks since i can see my peer disappearing from dashboard and following files/logs are written...

unclean_shutdown_dns.txt with some strange id inside..

2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.168:53: read udp 192.168.25.40:59355->192.168.40.168:53: i/o timeout
2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.68:53: read udp 192.168.25.40:59354->192.168.40.68:53: i/o timeout
2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:262: Upstream resolving is Disabled for 30s

Something is definitely wrong with communication after 0.28.0 update please take look at this issue... 😢

lixmal commented 3 months ago

Can you provide a netbird status -dA please? Even better a complete debug bundle netbird debug for 1m -A

nikolicjakov commented 3 months ago

netbird.debug.3041785520.zip

@lixmal Here you go...

yaneony commented 3 months ago

Having same issue here.

mlsmaycon commented 3 months ago

@yaneony @mrota83 @MDMeridio001 are you running on windows? can you share the output requested here: https://github.com/netbirdio/netbird/issues/2157#issuecomment-2179963612 ?

nikolicjakov commented 3 months ago

Thank you guys for new release everything looks to be working fine in v0.28.1

MDMeridio001 commented 3 months ago

@mlsmaycon No, the client is an Ubuntu Server machine. I have updated to 0.28.1 but I'm still facing the same issue. Here is the output of netbird debug for 1m -A: netbird.debug.80462564.zip

nikolicjakov commented 3 months ago

What did you update client or server components to new v0.28.1 version?

MDMeridio001 commented 3 months ago

@nikolicjakov both of them.

MDMeridio001 commented 3 months ago

Just an update, I have also tried updating both the server components and the client to version 0.28.2 and the issue persists.

Ryan00793 commented 3 months ago

This is not fixed in 0.28.3 hosts that previously had the issue however, on a fresh install of ubuntu 22.04 it appears to be working. Luckily for me this is being used in a VM that is only used for netbird so wasn't worried about other things.

ykorzikowski commented 3 months ago

Maybe related to this https://github.com/netbirdio/netbird/issues/2219

I start noticing a really shaky server environment after I updated my server-netbird-clients (dns issues, packet loss).

Did not do any further debugging and downgraded all clients to 0.27.10 instead, looks stable again.

Edit:

Some minds: I notice i can dig&nslookup my custom dns server and the local netbird dns. But a curl results in a can not resolve error. This issue was not persistant (see my macbook dns firewall issue). Maybe its a iptables related issue? Some packages got filtered by a wrongly configured rule?