[SELF-HOSTED] Wrong public IP for client on a VPS

[kalwadi]

Describe the problem

NetBird Client (Debian 12 server hosted on Hetzner) shows wrong IP as Public IP. 172.21.0.1 is shown as Public IP Wrong IP is sometimes shown with docker client as well

To Reproduce

Steps to reproduce the behavior:

1. Create VPS on Hetzner (debian 12)
2. Install native netbird client
3. Join network
4. Assigned Public IP is 172.21.0.1

Expected behavior

Actual Public IP is shown

Are you using NetBird Cloud?

No, Self-host NetBird

NetBird version

0.27.5

Additional context https://github.com/netbirdio/netbird/issues/1728#issuecomment-2105639940

My controller is running on Hetzner VPS (arm CPU) DNS record for subdomain is directly pointed to this IP Authentik is used for authentication

Running NetBird client on Debian 12 VM (in Proxmox) at home shows correct public IP

On a clean VPS (no docker installed) I have tried running native client and wrong IP was showing

After that I installed docker and created container with this docker-compose.yml file, correct IP was showing in dashboard

volumes:
 netbird-client:

services:
  netbird:
    image: netbirdio/netbird:latest
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_RESOURCE
    environment:
      - NB_SETUP_KEY=xxxx
      - NB_MANAGEMENT_URL=https://netbird.my-domain.tld:33073
    volumes:
      - netbird-client:/etc/netbird
    restart: always

Some findings after a bit of testing:

• New VPS (tried in 3 different Hetzner locations) 2x ARM CPU, 1x x86 CPU

  • Native client shows 172.21.0.1 as public IP
  • docker client with above shown docker-compose.yml shows correct IP

• VPS where I run my NetBird controller shows wrong IP with native client and docker client

  • only thing running on this VPS is the controller (and broken netbird client)
  • this VPS was created for this and no other things were setup on this before

[nuterum]

To make sure we speak of the same thing i will describe what i understand and explain what happen.

If i understand correctly you have install netbird client on the same server as netbird server (self-hosted). In my case it what i have done.

Then in this case you find a strange IP (in fact local IP) in your netbird peer.

It is not really a bug in itself but a problem with the understanding of public IP. The "Public IP" show in the peer is the IP that is use to join the server in the IP header when connecting.

Normally you will pass by internet that why the "Public IP" is correct but in case of local machine or even local network because you can use netbird in local network too then the IP on the IP header will be a local one and will not have been change by a NAT.

That why in my case my server having netbird client show a public IP of "172.19.0.1". The only downside is that without going to change the "location_country_code text,location_city_name text,location_geo_name_id integer" manually in the sqlite db then my peer will be unknow for region.

The only thing not test for now is another netbird client on another VPS of the same provider.

To test further i may need the configuration of your netbird server because depending on how its configure and how the provider handle VPS maybe the VPS exchange packet without going on internet but by local network of the provider (even if in different location it can be connect together by virtual network) and in this case it will depend of the configuration of your server but i guess your self-hosted netbird is a docker using the network 172.21.0.0/16 and the server have an interface "br-XXXXX" with ip "172.21.0.1/16" and local network NAT the source IP to 172.21.0.1.

You can at least check what ip is resolve for "netbird.my-domain.tld" on your different VPS with or without docker before posting your configuration.

If my guess is right docker may not use the DNS of the provider and find the public IP of "netbird.my-domain.tld" then pass by internet and without docker we use the provider DNS that answer a local IP for "netbird.my-domain.tld".

But to realize this test i need your server configuration and to know if you domain is register on Hetzner or not to be able to reproduce the same environement.

[xrehpicx]

Is there any way i can manually set the peer public IP, or whatever ip i want the peer to advertise as it's public IP

[remogeissbuehler]

I had a smilar issue where some of my clients showed up as having the docker gateway IP (172.17.0.1 in this example) as their public IP (cf. #1728). I looked at the Caddy logs and saw that the X-Forwarded-For Headers were also set to this IP (172.17.0.1).

I checked the upstream caddy issues and this appears to occur when clients connect via IPv6 (see this issue: https://github.com/caddyserver/caddy/issues/4339). This happens since docker containers by default don't communicate over IPv6, so the docker gateway "NATs" these connections which then have the gateway address as a source.

I added IPv6 to the docker setup, by adding the following to /etc/docker/daemon.json

{
    "experimental": true,
    "ip6tables": true,
    "default-address-pools": [
        { "base": "172.17.0.0/16", "size": 16 },
        { "base": "172.18.0.0/16", "size": 16 },
        { "base": "172.19.0.0/16", "size": 16 },
        { "base": "172.20.0.0/14", "size": 16 },
        { "base": "172.24.0.0/14", "size": 16 },
        { "base": "172.28.0.0/14", "size": 16 },
        { "base": "fd42:1234:5678::/48", "size": 64 }
    ]
}

Note that adding address pools isn't strictly necessary, you can also set them on container creation (i.e. in the docker-compose file in this scenario). I used an IPv6 ULA, but I'm sure there's other / better ways ;)

We also need to enable IPv6 networking in the docker compose file:

networks:
  netbird:
    enable_ipv6: true

With those changes, the peers show up correctly with their IPv6 listed as the public IP.

[cipherw0lf]

I'm having the same issue. I used a Ubuntu 22.04 VPS and simply ran (as root) the getting-started-with-zitadel.sh script as per the self-host quickstart guide.

Relevant info:

• Im not using IPv6
• Im not running any clients on the server
• ISSUE: All Public WAN addresses show the docker IP address of Caddy container instead of actual WAN address of the clients
• I do believe the WAN address was showing correctly immediately after deployment (running script)
• I attempted to update the server recently when v0.28.7 dropped
• Issued commands as root:
  • docker compose down
  • docker compose pull
  • docker compose up -d
• After this i believe issue started occurring
• No other options were changed
• All firewalls off

Note: the VPN is still working and connectivity doesn't seem to be affected as this point.

Caddy log snippet:

DBG ts=1723207925.2943292 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=management:80 duration=0.072324231 request={"remote_ip":"172.18.0.1","remote_port":"33864","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"netbird.MYDOMAIN.net","uri":"/api/routes","headers":{"Sec-Gpc":["1"],"Authorization":["REDACTED"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-Host":["netbird.MYDOMAIN.net"],"X-Forwarded-For":["172.18.0.1"],"Cookie":["REDACTED"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"],"Dnt":["1"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Sec-Ch-Ua-Mobile":["?0"],"Content-Type":["application/json"],"X-Forwarded-Proto":["https"],"Priority":["u=1, i"],"Accept-Language":["en-US,en-ZA;q=0.9,en;q=0.8,th;q=0.7"],"X-Real-Ip":["172.18.0.1"],"Sec-Fetch-Dest":["empty"],"Referer":["https://netbird.MYDOMAIN.net/peers"],"Accept":["application/json"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"netbird.MYDOMAIN.net"}} headers={"Vary":["Origin"],"Date":["Fri, 09 Aug 2024 12:52:05 GMT"],"Content-Length":["1425"],"Content-Type":["application/json; charset=UTF-8"]} status=200