[SELF-HOSTED] Geographically dispersed peers get the exact same "Public IP" - And the "Public IP" is actually a Private IP on the 10.x.x.x subnet!

[pomology]

The problem I'm having is that all my peers are getting the very same "Public IP" which is 10.89.1.53. The "Public IP" is actually a private IP, and every single peer gets exactly the same "Public IP" address! Needless to say, none of my peers can talk to each other, nor ping each other, etc, etc.

The peers are on widely geographically dispersed networks, and yet they all have the same problem.

Here is my Caddyfile, from NetBird's default configs:

{
  debug
    servers :80,:443 {
    protocols h1 h2c
  }
}

(security_headers) {
    header * {

        Strict-Transport-Security "max-age=3600; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        X-XSS-Protection "1; mode=block"
        -Server
        Referrer-Policy strict-origin-when-cross-origin
    }
}

:80, REDACTED.REDACTED.com:443 {
    import security_headers
    reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
    reverse_proxy /api/* management:80
    reverse_proxy /management.ManagementService/* h2c://management:80
    reverse_proxy /* dashboard:80
}

And here's my compose.yml, again just built on NetBird's default:

version: "3"
services:
 # Caddy reverse proxy
  caddy:
    image: caddy:latest
    restart: unless-stopped
    networks:
      - netbird
    ports:
      - '443:443'
      - '80:80'
    volumes:
      - netbird_caddy_data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile

  #UI dashboard
  dashboard:
    image: netbirdio/dashboard:latest
    restart: unless-stopped
    networks:
      - netbird
    environment:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://REDACTED.REDACTED.com:443
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://REDACTED.REDACTED.com:443
      # OIDC
      - AUTH_AUDIENCE=REDACTED
      - AUTH_CLIENT_ID=REDACTED
      - AUTH_CLIENT_SECRET=
      - AUTH_AUTHORITY=https://REDACTED.okta.com
      - USE_AUTH0=false
      - AUTH_SUPPORTED_SCOPES=openid profile email
      - AUTH_REDIRECT_URI=/auth
      - AUTH_SILENT_REDIRECT_URI=/silent-auth
      - NETBIRD_TOKEN_SOURCE=idToken

  # Signal
  signal:
    image: netbirdio/signal:latest
    restart: unless-stopped
    networks:
      - netbird

  # Management
  management:
    image: netbirdio/management:latest
    restart: unless-stopped
    networks:
      - netbird
    depends_on:
      - dashboard
    volumes:
      - netbird-mgmt:/var/lib/netbird
      - ./management.json:/etc/netbird/management.json
    command: [
      "--port", "80",
      "--log-file", "console",
      "--log-level", "info",
      "--disable-anonymous-metrics=false",
      "--single-account-mode-domain=REDACTED.REDACTED.com",
      "--dns-domain=netbird.selfhosted",
      "--idp-sign-key-refresh-enabled"
]

  # Coturn
  coturn:
    image: coturn/coturn:latest
    restart: unless-stopped
    domainname: REDACTED.REDACTED.com
    volumes:
      - ./turnserver.conf:/etc/turnserver.conf:ro
    network_mode: host
    command:
      - -c /etc/turnserver.conf

volumes:
  netbird-mgmt:
  netbird_caddy_data:

networks:
  netbird:

[lixmal]

If your setup is behind another proxy (cloudflare etc) you might need to accept X-Forwarded-For headers from there to display the real public IP. See https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults

As for

Needless to say, none of my peers can talk to each other, nor ping each other, etc, etc.

Could you please provide a netbird status -dA from one of the peers? I suspect they are not connecting to each other

[pomology]

Thanks! My server is running on AWS EC2, so not behind any other proxy. Just to be sure, I tried accepting X-Forwarded-For headers in my Caddyfile, but still got the same result....

Here's a netbird status -dA for one of the peers:

Peers detail:
 ip-172-45-16-167.connect.anon-IbY1H.domain:
  NetBird IP: 100.120.11.112
  Public key: REDACTED
  Status: Disconnected
  -- detail --
  Connection type: 
  Direct: false
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

OS: darwin/arm64
Daemon version: 0.27.5
CLI version: 0.27.5
Management: Connected to https://connect.anon-IbY1H.domain:443
Signal: Connected to https://connect.anon-IbY1H.domain:443
Relays: 
  [stun:connect.anon-IbY1H.domain:3478] is Unavailable, reason: dial: failed to listen: dial: dial udp: lookup connect.anon-IbY1H.domain: no such host
  [turn:connect.anon-IbY1H.domain:3478?transport=udp] is Unavailable, reason: create client: lookup connect.anon-IbY1H.domain: no such host
Nameservers: 
FQDN: rmacbook-pro.connect.anon-IbY1H.domain
NetBird IP: 100.120.67.54/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 0/1 Connected

[lixmal]

Thanks! My server is running on AWS EC2, so not behind any other proxy. Just to be sure, I tried accepting X-Forwarded-For headers in my Caddyfile, but still got the same result....

Do you run a loadbalancer in front (ELB/ALB or some ingress controller)? That would have the same result

Here's a netbird status -dA for one of the peers:

It seems there is some issue with the stun/turn server connection Do you happen to have the same domain for netbird internally as well as for management/signal etc?

[pomology]

So it appears that this only occurs when using Podman. I just now tried rootfull docker instead and the issue goes away, I get correct external IP's that display normally and clients can ping each other. Is Netbird known to be incompatible with Podman? Any idea what's going on?

[pomology]

For testing purposes, I just rebuilt the whole thing with rootless docker, and I get the same issue of an internal IP being shown where the External IP should be, and no communication between peers. It seems these configs simply don't work for operating in a rootless container.

Rootfull containers seem to work fine, but running rootfull is arguably very dangerous.

What do you guys recommend for running rootless? Any recommended configs? Thank you!

[pomology]

Per the below website, it appears this is expected behavior.

https://rootlesscontaine.rs/how-it-works/netns/incoming/

There are apparently ways to get around it, but quite complicated. I hope the devs can work toward as much rootless implementation as possible, to further improve security.