search

netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
11.17k stars 515 forks source link

After netbird up, server can not be accessed by public IP. #2638

Open hwiorn opened 1 month ago

hwiorn commented 1 month ago

Describe the problem

I'm netbird newbie. I installed self-hosted version. this is my infra setup.

  1. Some devices are in private area, home and office usually: netbird works nicely.
  2. Some servers are on vultr hosting.
    • it has public IP and domain.
    • netbird works nicely too, but It makes accessing public IP and domain unusable.

Before netbird up(or after netbird down).

After netbird up.

How can I enable public IP accessing? To Reproduce

Steps to reproduce the behavior:

  1. Install self-hosted netbird.
  2. create new small instance named "server A" on vultr.
  3. Install latest netbird client on server A.
    • curl -fsSL https://pkgs.netbird.io/install.sh | sh
  4. netbird up
    • netbird up --management-url https://vpn.mysite.com:33073 --setup-key XXXXX
  5. Check peer is alive on dashboard.
  6. Access server-a.netbird.selfhosted and check it's okay.
  7. Access public IP and domain of server A.

Expected behavior

Can access public IP and domain of server A.

Are you using NetBird Cloud?

No. I use self-host netbird.

NetBird version

0.29.4

NetBird status -dA output:

Peers detail:
 jjjj-mbp16.anon-zDCem.domain:
  NetBird IP: 100.120.252.2/32
  Public key: WvzlfOaOdvoEL58I2G65RF6YDrPZy9GGskCxaAR29jc=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 kkkk-1.anon-zDCem.domain:
  NetBird IP: 100.120.57.134/32
  Public key: APLRm4PzviXz/my6zL7W83EbakpntQMeH3Hr1/sDtwg=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 hhhh.anon-zDCem.domain:
  NetBird IP: 100.120.6.30/32
  Public key: 0QMKayEh1Zotxd0fjkG9PY4TuTzDHR2+tOdrfmFEQnQ=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 iiii.anon-zDCem.domain:
  NetBird IP: 100.120.148.94/32
  Public key: /vIMf0EL8Iuus3UR4d4S18315aKhc9ipxXIzBfq2smk=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 ralabdev.anon-zDCem.domain:
  NetBird IP: 100.120.1.200
  Public key: RqmwwDMD7jo647NfyREx+3DmbDYvGgxUQLKANVbm/jw=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 dev1.anon-zDCem.domain:
  NetBird IP: 100.120.29.48
  Public key: EiVVdbQBc4sljZsWf5XZUtX8ncWgnPI6DeYA+Z+kBTc=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 vpn.anon-zDCem.domain:
  NetBird IP: 100.120.41.169
  Public key: c/QRxOPEpimFN3H8VJaAA1r6hOpP7k7IRMQwUnpUonk=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:50340
  Relay server address: rel://vpn.mycoolsite.com:33080
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: 1 minute ago
  Transfer status (received/sent) 2.4 KiB/2.4 KiB
  Quantum resistance: false
  Routes: 0.0.0.0/0
  Latency: 3.32751ms

 timlee-gram16z.anon-zDCem.domain:
  NetBird IP: 100.120.102.36
  Public key: 4uwK3BhnBXYTbo3DmBdkM2pPUHiwlfRxm2u6c8sBDyo=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rel://vpn.mycoolsite.com:33080
  Last connection update: 25 seconds ago
  Last WireGuard handshake: 1 minute ago
  Transfer status (received/sent) 272 B/484 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 gpu-aaaa.anon-zDCem.domain:
  NetBird IP: 100.120.134.128
  Public key: wI2lezpk2x71GV74SAOBtjrvali17kPRfgMI1qNfTGY=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.2:51820
  Relay server address: rel://vpn.mycoolsite.com:33080
  Last connection update: 1 minute, 4 seconds ago
  Last WireGuard handshake: 1 minute ago
  Transfer status (received/sent) 336 B/484 B
  Quantum resistance: false
  Routes: -
  Latency: 3.472606ms

 iphone-tim.anon-zDCem.domain:
  NetBird IP: 100.120.170.194
  Public key: PjRylF2HjANwkl5HuHq2/eUDVB5MahoYmCh0a9tK5Rs=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 pms.anon-zDCem.domain:
  NetBird IP: 100.120.194.176
  Public key: QrlGRYd5tieJzHE0eZSB9pqrWuD6Z6nlTkTXDm/6vQo=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 tim-macbook-pro-m2-14inch.anon-zDCem.domain:
  NetBird IP: 100.120.199.90
  Public key: 7knQk+6HfozpqJuUVWG+6F2HvYoOPGt9ESfYzAHX/0I=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rel://vpn.mycoolsite.com:33080
  Last connection update: 25 seconds ago
  Last WireGuard handshake: 1 minute ago
  Transfer status (received/sent) 14.4 KiB/10.0 KiB
  Quantum resistance: false
  Routes: -
  Latency: 0s

 gpu-llll.anon-zDCem.domain:
  NetBird IP: 100.120.251.81
  Public key: WiRm++U86sBSdtNtwnwqwm/SAlUkNc3VOn4YUGPLdBs=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

OS: linux/amd64
Daemon version: 0.29.4
CLI version: 0.29.4
Management: Connected to https://vpn.anon-R8VTi.domain:33073
Signal: Connected to http://vpn.anon-R8VTi.domain:10000
Relays:
  [stun:vpn.anon-R8VTi.domain:3478] is Unavailable, reason: stun request: context deadline exceeded
  [turn:vpn.anon-R8VTi.domain:3478?transport=udp] is Unavailable, reason: allocate: all retransmissions failed for Gb1263WLrQqMZd8Q
  [rel://vpn.anon-R8VTi.domain:33080] is Available
Nameservers:
  [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: server-a.anon-zDCem.domain
NetBird IP: 100.120.194.194/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 4/13 Connected

Do you face any (non-mobile) client issues?

Please provide the file created by netbird debug for 1m -AS. netbird.debug.375981251.zip

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

SSH always got operation timed out error, when netbird is connected.

ssh  -p 20220 suser@158.247.XXX.YYY
ssh: connect to host 158.247.XXX.YYY port 20220: Operation timed out
hwiorn commented 1 month ago

I figure out the issue was caused by exit-node. After exit-node routing was disabled on server side, this issue has gone. On the other hands, disabling it only on client side doesn't work.

But, the exit-node feature is I wanted to use. I don't know what settings cause this issue.

I found another issue is, network transmission is really slow when exit-node routing is active, even both vpn and device in the same network.

bravosierrasierra commented 1 month ago

try to add inside netbird catch-all DNS service via google/cloudflare. I have same problem and this strange solution works. And it works for fix strange internal dns servers resolving problems

hwiorn commented 1 month ago

Do you mean "Nameservers" in netbird dashboard? If it is what your talking about, I already did use google DNS in this netbird network. I think I could test to toggle DNS settings once a more.

lixmal commented 1 month ago

@hwiorn Can you confirm if the server with the public IP is the routing client (part of distribution group for the exit node) or the routing server (the exit node itself)?

According to your netbird status it seems to be the former. In that case all responses (e.g. to your ingress ssh attempt) are routed via the exit node and become inaccessible from elsewhere unless there is another more specific route installed.

hwiorn commented 1 month ago

@lixmal I thought exit node peers can be by installing netbird client. I didn't realize it has ingress issue. I installed the netbird client to make an exit node within my VPN server which was already netbird self-hosted. I understand this behavior that you said if exit node is active. But, I don't get it how I can make the routing to accept external access from public IP using netbird. Do I have to set some internal routing between wg IPs in netbird? Or just settings up "the routing server" is enough? And, I can't figure out what's the routing server in your comment.