Open Enailis opened 4 months ago
Hi @Enailis,
how exactly is the route to 10.10.10.1 set up? Are you sure the configured routing peer is online and successfully connected to the users peer that tries to ping? Is that connection direct or relayed? So with netbird status -d
can you detect a difference in the connection when it is working compared to when it is not working?
Hi @pascal-fischer,
We have a network route to 10.10.10.1/32 using our internal servers as peer group. All servers in this group have access to 10.10.10.1. This route is distributed to all users. We have 3 different peers in this group, they're all online. The servers can't ping the users with their Netbird's IP. The users can ping the servers using their real IP but not their Netbird's IP. The connection to the 3 servers in the peer group is relayed.
We actually can't detect any difference in the netbird status -d
when it's working and when it's not. The current configuration gives me this result for netbird status -d
, there is other peers but they all look the same as the one shown here:
server.mycompany.com:
NetBird IP: 100.73.252.226
Public key: RupexIsExt4J2oKsN4avstKkjD03vlSq728BzT/uvB8=
Status: Connected
-- detail --
Connection type: Relayed
Direct: false
ICE candidate (Local/Remote): relay/prflx
ICE candidate endpoints (Local/Remote): 90.90.90.90:63293/80.80.80.80:63293
Last connection update: 2024-04-22 13:58:48
Last WireGuard handshake: 2024-04-22 14:11:34
Transfer status (received/sent) 1.9 KiB/1.5 KiB
Quantum resistance: false
Routes: -
Latency: 55.684815ms
Daemon version: 0.27.3
CLI version: 0.27.3
Management: Connected to https://vpn.mycompany.com:33073
Signal: Connected to http://vpn.mycompany.com:10000
Relays:
[stun:vpn.mycompany.com:3478] is Available
[turn:vpn.mycompany.com:3478?transport=udp] is Available
Nameservers:
[10.10.10.1:53] for [gitlab.mycompany.com] is Available
FQDN: ena.mycompany.com
NetBird IP: 100.73.213.219/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 8/13 Connected
Even if everything looks fine, I cannot access gitlab.mycompany.com
.
To add something from my original issue, it now works for some windows users. The client takes a long time to connect and sometimes users have to do netbird up
/netbird down
multiple times before it actually works. It still doesn't work for Linux, macOS and some windows users.
Hello, I'm working on the same Netbird instance as @Enailis
Some corrections / additional information on the above post:
Here is some other additional information:
On Windows clients (our users connected with SSO), our nameserver (10.10.10.1:53) is unstable, and its availability can change from one netbird down & netbird up
to another for no apparent reason.
When it's available, we can access gitlab.mycompany.com
and server.mycompany.vpn
without any problem.
However, when it's unavailable ([10.10.10.1:53] for [gitlab.mycompany.com] is Unavailable, reason: 1 error occurred: * read udp 192.168.1.182:53408->10.10.10.1:53: i/o timeout)
), we can no longer access gitlab.mycompany.com
but we can still access server.mycompany.vpn
.
On our Linux clients (other users connected with SSO), other behaviors appear.
Our nameserver (10.10.10.1:53) is always marked as available in a netbird status -d
, however, it is impossible to access gitlab.mycompany.com
or server.mycompany.vpn
Here is a client's /etc/resolv.conf
file:
# Generated by NetworkManager
nameserver 192.168.1.1
If I run dig gitlab.mycompany.com
, I don't get an IP address back. However, if I run dig @10.10.10.1 gitlab.mycompany.com
, its IP appears. So by adding the line nameserver 10.10.10.1
in the clients' /etc/resolv.conf
files, we can access our gitlab but we can't still access server.mycompany.vpn
.
Note that we can still access our gitlab via its IP address (the IP given by Netbird and its real IP). Our routes are therefore well configured, the problem only comes from DNS resolution.
Finally, note that this problem never appears for Linux clients installed with a Setup Key (our servers). Here's their /etc/resolv.conf
file:
...
nameserver 127.0.0.53
options edns0 trust-ad
search company.vpn company.com
We therefore believe that the problem only comes from Netbird clients, which cannot apply DNS configurations to our workstations (Linux and Windows).
Hello, here is some additional information about our Windows client errors.
Here are the lines in the client.log file when the error [10.10.10.1:53] for [gitlab.mycompany.com] is Unavailable, reason: 1 error occurred: * read udp 192.168.1.182:53408->10.10.10.1 :53: i/o timeout)
appears on our Windows clients:
2024-04-25T11:58:42+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: 1 error occurred:
* executing dial hook: 1 error occurred:
* adding route reference: failed to add route for prefix 90.90.90.90/32: add route to table: PowerShell add route: exit status 1
2024-04-25T11:58:43+02:00 ERRO util/net/listener_generic.go:128: Error executing listener write hook: adding route reference: failed to add route for prefix 90.90.90.90/32: add route to table: PowerShell add route: exit status 1
Similar issue here on macOS.
Server: 192.168.99.1
Address: 192.168.99.1#53
Non-authoritative answer: Name: docker.my-localdomain.local Address: 192.168.99.125
OS: darwin/arm64 Daemon version: 0.27.10 CLI version: 0.27.10 Management: Connected to https://netbird.mydomain.com:33073 Signal: Connected to http://netbird.mydomain.com:10000 Relays: [stun:netbird.mydomain.com:3478] is Available [turn:netbird.mydomain.com:3478?transport=udp] is Available Nameservers: [192.168.99.1:53] for [my-localdomain.local, mydomain.com] is Unavailable, reason: 1 error occurred:
Describe the problem
We're using the self-hosted version of Netbird and everything is setup according to the documentation. Sometimes the custom nameserver is resolved, sometimes it isn't. That's without ever touching the config on the web interface.
To give more context here is our network configuration:
When a user is connected to the Netbird VPN, he can ping every server and every user without any problem. For example, users can ping Gitlab's Netbird IP:
But users cannot ping pfSense's DNS Resolver IP:
The
netbird status -d
command returns this problem:Apart from this, we have no logs server side about this, and only
i/o timeout
for the pfSense's DNS Resolver IP invar/log/netbird/client.log
.But sometimes, without changing anything, either client or server side, everything works just fine.
This issue appears on every OS: Windows 11, macOS 14.4.1 (23E224) and Ubuntu 22.04.
To Reproduce
Since the problem is random, we have no clue how to reproduce this problem.
Expected behavior
The Nameserver is supposed to be constantly recognize by Netbird without being randomly unavailable.
Are you using NetBird Cloud?
We're using Netbird self-hosted solution.
NetBird version
Every user is up to date: 0.27.3.