NGINX reverse proxy question

[andreklug]

Hi, I am trying to reverse proxy netbird to a local installation, but always run into "Bad Gateway". The setup is as follows:

Firewall: Ports 443 and 80 (TCP/UDP) forwarded to a VM (192.168.1.2) that runs Nginx (not Nginx proxy manager). Several other services are working through the proxy. The config for Netbird listens to nb.mydomain.com

Netbird is installed installed with authentik (auth.mydomain.com). I verified that this is working in general by temporary hosting netbird on a VPS at my hoster and created another application and provider in Authentik that points to nb.mydomain.com. The netbird VM has 192.168.1.3

nginx-site.conf.txt docker.compose.yml.txt setup2.env.txt

I am sure I am missing something, but what?

EDIT: I have changed the ports in the nginx config as follows:

upstream dashboard {
    server 192.168.10.97:80;
    keepalive 10;
}

upstream signal {
    server 192.168.10.97:10000;
}

upstream management {
    server 192.168.10.97:33073;
}

Now I get this:

Thank you in advance!

[lixmal]

Have you taken a look at this section https://docs.netbird.io/selfhosted/selfhosted-guide#configuration-for-your-reverse-proxy in the self-hosted guide?

[andreklug]

Yes, followed it to the letter but that didn't help. The auth issue is solved, in fact that was mixed up with my first test of Netbird on a hosted VPS. I have re-taken the authentik steps and this seems to work now. I am getting to load the dashboard and log in sucessful, see teh download instructions. but cannot connect to the vpn.

There are some questions the documentaiton doesn't answer:

• which hostname do I enter in setup.env? nb.mydomain.com does not resolve to the netbird VM but to the gateway. I did edit /etc/hosts on the netbird vm so that nb.mydomain.com points to localhost in case it is needed.
• what to fill in for NETBIRD_MGMT_API_ENDPOINT in setup.env?

[andreklug]

so, here's the current setup:

./configure output:

Letsencrypt was disabled, the Https-endpoints cannot be used anymore
 and a reverse-proxy with Https needs to be placed in front of netbird!
The following forwards have to be setup:
- https://nb.mydomain.com:443 -http-> dashboard:80
- https://nb.mydomain.com:33073/api -http-> management:33073
- https://nb.mydomain.com:33073/management.ManagementService/ -grpc-> management:33073
- https://nb.mydomain.com:81/signalexchange.SignalExchange/ -grpc-> signal:80

nginx:

server {
    listen 443 ssl http2;
    server_name nb.mydomain.com;

    listen 444 ssl http2;
    server_name nb.mydomain.com;

    listen 81 ssl http2;
    server_name nb.mydomain.com;

    listen 33073 ssl http2;
    server_name nb.mydomain.com;

    # This is necessary so that grpc connections do not get closed early
    # see https://stackoverflow.com/a/67805465
    client_header_timeout 1d;
    client_body_timeout 1d;

    ssl_certificate /etc/letsencrypt/live/nb.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/nb.mydomain.com/privkey.pem;

    ssl_protocols TLSv1.3;  # Add this line

    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Scheme $scheme;
    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forwarded-Host $host;
    grpc_set_header         X-Forwarded-For $proxy_add_x_forwarded_for;

    access_log /var/log/nginx/nb.access.log;
    error_log /var/log/nginx/nb.error.log;

    ssl_verify_client off;

    # Proxy dashboard
    location / {
        proxy_pass http://192.168.10.97:80;
    }

    # Proxy Management http endpoint
    location /api {
        proxy_pass https://192.168.10.97:33073;
    }

    # Proxy Management grpc endpoint
    location /management.ManagementService/ {
        grpc_pass grpc://192.168.10.97:33073;
        grpc_ssl_verify off;
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }

    # Proxy Signal
    location /signalexchange.SignalExchange/ {
        grpc_pass grpc://192.168.10.97:80;
        grpc_ssl_verify off;
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }

}

[sgtaziz]

I also had issues with using NGINX as my reverse proxy. Got it working with below configuration:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name netbird.mydomain.com;

    #Use letsencrypt.org to get a free and trusted ssl certificate
    ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem; # managed by Certbot

    error_log /var/log/nginx/netbird.error.log error;
    access_log /var/log/nginx/netbird.access.log;

    # This is necessary so that grpc connections do not get closed early
    # see https://stackoverflow.com/a/67805465
    client_header_timeout 1d;
    client_body_timeout 1d;

    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Scheme $scheme;
    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forwarded-Host $host;

    # Proxy dashboard
    location / {
        proxy_pass http://dashboard-upstream;
    }
    # Proxy Signal
    location /signalexchange.SignalExchange/ {
        grpc_pass grpc://signal-upstream;
        # Can't remember why I turned this off, but it works fine without it
        #grpc_ssl_verify off; 
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }

    # Proxy Management http endpoint
    location /api {
        proxy_pass http://management-upstream;
    }

    # Proxy Management grpc endpoint
    location /management.ManagementService/ {
        grpc_pass grpc://management-upstream;
        #grpc_ssl_verify off;
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }
}

[andreklug]

Thank you @sgtaziz !

Are you running netbird on the same machine as nginx? I don't see the definitions for the location endpoints. Also, did you in the setup.env set these?

NETBIRD_MGMT_API_PORT="444"

NETBIRD_MGMT_API_ENDPOINT="nb.mydomain.com" NETBIRD_SIGNAL_PORT="81"

EDIT: And which ports did you forward to nginx?

[sgtaziz]

Yes, I'm running it on the same machine. The endpoints are configured at the top of my nginx site file as such:

upstream dashboard-upstream {
    server 127.0.0.1:8086;
    keepalive 10;
}
upstream signal-upstream {
    server 127.0.0.1:8087;
}
upstream management-upstream {
    server 127.0.0.1:6443;
}

And the related setup.env:

NETBIRD_MGMT_API_PORT=6443
NETBIRD_SIGNAL_PORT=8087

Also, I did modify the docker-compose.yml manually, since there is no env variable to change the dashboard port:

services:
  dashboard:
    image: netbirdio/dashboard:latest
    restart: unless-stopped
    ports:
      - 8086:80 # 8086 since 80 is already used by nginx
      #- 443:443
    environment:
  ....

Nginx ports open are only 80/tcp and 443/tcp. I do have the COTURN related ports open, but that's not through NGINX, just opened it from the docker container and my firewall.

Feel free to change the ports in setup.env and the dashboard port in docker-compose.yml file to whatever you need, just make sure its reflected in the nginx upstream configuration.

EDIT: Also, I didn't have grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for; in my original file. But, definitely keep it there for yours, it fixed an issue for me where clients did not have a public IP configured. So thank you! 😄

[Xentec0]

Hi @sgtaziz, I'm running the same setup as you i believe. NGINX (as reverse proxy) and Netbird on the same machine. However, I can't get this to work as I'm stuck on an infinite loading screen on https://netbird.mydomain.com/peers, after logging in via authentik. My nginx_netbird.conf ist the same as yours and I've set up the same ports in my setup.env. Firewall / NAT definitley isn't the issue here. Would you mind sharing your docker-compose.yml please, so i can make sure I'm all good there?

[cannonfodda]

Hi @Xentec0. It might not be an issue any more but I was banging my head against this issue for a day or so. I noticed today that the configure script is appending the value of NETBIRD_MGMT_API_PORT to NETBIRD_MGMT_API_ENDPOINT and NETBIRD_MGMT_GRPC_API_ENDPOINT in the generated docker-compose file.

So if you use the settings from @sgtaziz above then NETBIRD_MGMT_API_PORT is set to https://:6443 which results in a timeout as the JS client can't communicate with this directly.

However, since we are proxying that connection via nginx if you set NETBIRD_MGMT_API_PORT=443 then it's handled by nginx and works for me. Check the network tab in the chrome/firefox debugging tools and if you are seeing that the client can't contact the management server then this may also be your problem.

Worked for me at least . I do wish there was an alternative deployment option to docker though, it's a pretty opaque process.

[itsmejomijohn]

I also had issues with using NGINX as my reverse proxy. Got it working with below configuration:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name netbird.mydomain.com;

    #Use letsencrypt.org to get a free and trusted ssl certificate
    ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem; # managed by Certbot

    error_log /var/log/nginx/netbird.error.log error;
    access_log /var/log/nginx/netbird.access.log;

    # This is necessary so that grpc connections do not get closed early
    # see https://stackoverflow.com/a/67805465
    client_header_timeout 1d;
    client_body_timeout 1d;

    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Scheme $scheme;
    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forwarded-Host $host;

    # Proxy dashboard
    location / {
        proxy_pass http://dashboard-upstream;
    }
    # Proxy Signal
    location /signalexchange.SignalExchange/ {
        grpc_pass grpc://signal-upstream;
        # Can't remember why I turned this off, but it works fine without it
        #grpc_ssl_verify off; 
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }

    # Proxy Management http endpoint
    location /api {
        proxy_pass http://management-upstream;
    }

    # Proxy Management grpc endpoint
    location /management.ManagementService/ {
        grpc_pass grpc://management-upstream;
        #grpc_ssl_verify off;
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }
}

@sgtaziz can you please share two docker compose files of netbird and nginx proxy manager and also there env sample file