netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
11.25k stars 517 forks source link

Stuck on loading dashboard #2226

Open MichalMarchewka opened 4 months ago

MichalMarchewka commented 4 months ago

Describe the problem

A Netbird is stuck on loading dashboard.

To Reproduce

Steps to reproduce the behavior:

  1. Install and setup Netbird following this guide
  2. Try to sign in to Netbird. After putting credentials and MFA Netbird stucks.

Expected behavior

Netbirds successfully loads and https://netbird.REDACTED.net/peers content is reached

Are you using NetBird Cloud?

Self-hosted

NetBird version

0.28.3

NetBird status -d output:

If applicable, add the `netbird status -d' command output.

Screenshots

image

Additional context

zitadel-1     | time="2024-07-01T21:00:28Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=auth.users3
zitadel-1     | time="2024-07-01T21:00:37Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=auth.tokens
zitadel-1     | time="2024-07-01T21:00:38Z" level=warning msg="instance by host" caller="/home/runner/work/zitadel/zitadel/internal/query/instance.go:210" domain=netbird.REDACTED.net error="failed to connect to `host=zdb user=zitadel database=zitadel`: failed to receive message (context canceled)" host=netbird.REDACTED.net
zitadel-1     | time="2024-07-01T21:00:38Z" level=error msg="unable to set instance" caller="/home/runner/work/zitadel/zitadel/internal/api/http/middleware/instance_interceptor.go:60" error="unable to get instance by host netbird.REDACTED.net: failed to connect to `host=zdb user=zitadel database=zitadel`: failed to receive message (context canceled)" externalDomain=netbird.REDACTED.net origin="https://netbird.REDACTED.net"
zitadel-1     | time="2024-07-01T21:00:38Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.apps7
zitadel-1     | time="2024-07-01T21:00:51Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.restrictions2
zitadel-1     | time="2024-07-01T21:00:51Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.idp_login_policy_links5
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.org_domains2
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.password_complexity_policies2
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.actions3
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.system_features
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.idp_user_links3
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="unable to query current state" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/state.go:70" error="ERROR: could not obtain lock on row in relation \"current_states\" (SQLSTATE 55P03)" projection=auth.user_sessions
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="state already locked" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:462" projection=auth.user_sessions
Jadefalkner commented 4 months ago

sounds similar to the problem i faced? maybe it helps

https://github.com/netbirdio/netbird/issues/1699

leoboyerbx commented 4 months ago

Same problem here

snoopckuu commented 4 months ago

Having same issue with Google iDP in Safari. Dashboard won't load and after sometime redirects back to Google Auth window. However Works in Google Chrome and Edge.

mlsmaycon commented 4 months ago

@MichalMarchewka can you share the caddy logs?

mlsmaycon commented 4 months ago

Same problem here

with our quick start guide too? if so, can you share the caddy logs?

mlsmaycon commented 4 months ago

Having same issue with Google iDP in Safari. Dashboard won't load and after sometime redirects back to Google Auth window. However Works in Google Chrome and Edge.

thanks for reporting, we will validate the issue with Google as IdP. In the mean time, can you check if https://app.netbird.io works well for you?

snoopckuu commented 4 months ago

Having same issue with Google iDP in Safari. Dashboard won't load and after sometime redirects back to Google Auth window. However Works in Google Chrome and Edge.

thanks for reporting, we will validate the issue with Google as IdP. In the mean time, can you check if https://app.netbird.io works well for you?

Yep, works well.

MichalMarchewka commented 4 months ago

@mlsmaycon Please find logs attached caddy_logs.txt

joshuahigginson1 commented 4 months ago

Hi @mlsmaycon , it appears that we've got the same issue after upgrading from v2.3.0 to v2.4.1 of the Netbird Dashboard. Both Chrome and Safari - I was never convinced that we had our auth configured correctly.

Looks like an issue with the change in the Dashboard useRedirect hook.

--- v2.3.0

Logging in or refreshing the page would result in a 100ms ~ 2sec wait for the console to show the Google Accounts page. This will always appear after a 'timeout' appears in the Chrome console.

After authenticating to Google, we see the following logs:

GET https://<our-domain>/auth 404 (Not Found)
Checking to see if there is an authorization response to be delivered.
Potential authorization request  https://<our-domain>/auth ...
Delivering authorization response

Peers page then loads.

Refreshing the page restarts this cycle and loads the Google Accounts page again.

Note: Chrome and Safari both block third-party cookies which is probably why we have to re-authenticate every page refresh.

---v2.4.1

Logging in or refreshing the page would result in a 100ms ~ 2sec wait for the console to show the Google Accounts page. This will always appear after a 'timeout' appears in the Chrome console. The difference between two versions is that we have 36 counts of 'Third Party Cookie will be Blocked' before Google Accounts page appears.

After authenticating to Google, we see the following logs:

GET https://<our-domain>/auth 404 (Not Found)
Checking to see if there is an authorization response to be delivered.
Potential authorization request  https://<our-domain>/auth ...
Delivering authorization response
** GET https://<our-domain>/auth.txt 404 (Not Found) **

Peers page never loads. Eventually, Netbird takes us to the Google Accounts page again.

axlroden commented 4 months ago

I have the same issue after updating to latest image, and running a previously working netbird with google idp.

For some reason it works in firefox, but not in chrome or safari.

Downgrading to v2.3.0 dashboard resolves the issue..

mlsmaycon commented 4 months ago

@heisbrot can you have a look at this issue?

heisbrot commented 4 months ago

Hey @MichalMarchewka ,

can you check if your SSL is properly configured? I see some certificate errors in the logs. Was you domain publicly accessible before you started with the getting started guide?

@joshuahigginson1 @axlroden @snoopckuu Do you have some logs? Or maybe a HAR file for the network requests. You can also try clearing the cache and cookies of your browser and check if the issue persists. (Or opening incognito) Does rebuilding the container fix the issue? docker compose up -d --force-recreate dashboard

ylluminate commented 4 months ago

I had the same problem, but it seems to be resolved by adjusting the /etc/hosts file. Here's what worked for me:

# /etc/hosts
127.0.0.1       localhost

Initially, I had other hostnames defined before localhost, but this setup caused issues with Docker containers accessing the identity provider configuration. Simplifying the /etc/hosts file to include only localhost for IPv4 resolved the problem.

As we're all probably aware, the /etc/hosts file is used to map IP addresses to hostnames. It's common practice to include multiple hostnames on a single line, such as 127.0.1.1 hostname.domain.tld hostname localhost, to associate multiple hostnames with an IP address. However, this configuration appears to interfere with Docker's DNS resolution inside containers.

In this case, the management container was repeatedly failing to fetch the OIDC configuration due to connection refusals on 127.0.1.1. Changing the /etc/hosts file to the above did resolve this...

If this is what's affecting others as well, and to perhaps avoid similar issues in the future, it might be beneficial to document this behavior as a known issue or recommend best practices for /etc/hosts configurations in the setup guides. Additionally, it could be helpful to investigate why Docker's DNS resolution behaves differently with multiple hostnames and whether there are configuration changes that can make it more robust to different /etc/hosts setups.

I'm not a big Docker fan myself, so I'm not terribly inclined to dig into this any further myself.

MichalMarchewka commented 4 months ago

Hey @heisbrot Yes, I have now rebuilt it from scratch. Unfortunately, it still doesn't work. Is there a chance to install Netbird without Docker?

Editing /etc/hosts per ylluminate instruction doesn't work for me.

ylluminate commented 4 months ago

@MichalMarchewka that is unfortunate to hear. Had hoped that would work for you. I do agree though, having a non-Docker install path would be good. Docker is such a headache.

axlroden commented 4 months ago

management.txt dashboard.txt 2.4.0.har.txt Logs of management and dashboard, and har file on 2.4.0 dashboard.

I tried doing force-recreate as well. Tried without cache on multiple browsers. Only firefox ends up on a peer list, after several reloads. Others either keep refreshing indefinately, or shows the google accounts login screen.

heisbrot commented 4 months ago

Hey @axlroden

Thank you for the logs. Can you try with the following image netbirdio/dashboard:pr-403 (instead of latest) and see if the problem persists?

jeehoonkang commented 4 months ago

@heisbrot I had the same problem, and netbirdio/dashboard:pr-403 fixed the issue. Thanks!

joshuahigginson1 commented 4 months ago

@heisbrot Thanks you Eduard, PR-403 has solved our issue.

snoopckuu commented 4 months ago

Can we merge the fix into the main branch?

axlroden commented 4 months ago

can confirm netbirdio/dashboard:pr-403 solved the issue.

MichalMarchewka commented 4 months ago

Amazing stuff, it's working now!

Good job, thank you.

TheKayneGame commented 3 months ago

This issue has not been resolved for me with the PR.

My setup contains Traefik and followed this tutorial from JimsGarage's video.

i am facing the same /peers screen loop.

See netbird-logs.txt for my logs.

I am running it on a local machine and externally its connected through cloudflare.

git-day commented 3 months ago

@heisbrot I am also experiencing the same problem as @MichalMarchewka where the login works as expected, but the dashboard never loads.

My setup is new and have followed the advanced guide here https://docs.netbird.io/selfhosted/selfhosted-guide#requirements. The chosen Idp is Zitadel and have followed the guide here on setup https://docs.netbird.io/selfhosted/identity-providers#zitadel.

I've noticed that there are a number of issues on Github which have been raised that are of a similar theme. I too have tried the suggested netbirdio/dashboard:pr-403 image, but doesn't work.

Please let me know if you would like more info about my setup, config, logs or to support troubleshooting.

snoopckuu commented 3 months ago

Hello everyone,

Happy to confirm that dashboard latest version 2.5.0 released yesterday fixed issue for me!

Thank you!

git-day commented 3 months ago

@snoopckuu , can you please share the version update details?

snoopckuu commented 3 months ago

@snoopckuu , can you please share the version update details?

Sure, i am talking about this release that fixed the issue: https://github.com/netbirdio/dashboard/releases/tag/v2.5.0

git-day commented 3 months ago

@snoopckuu thanks. Is there an easy way to identify what version of the dashboard that has been deployed via docker, given that i can't load the dashboard?

git-day commented 3 months ago

@mlsmaycon can you please help point me in the right direction. How do i troubleshoot this issue? Is there a set of logs to reference and assess where the issue might be? At this stage I have nothing to go by.

heisbrot commented 3 months ago

Hey @git-day ,

you can run docker image inspect netbirdio/dashboard. Under Labels there should be the image version. Latest version as of today is 2.5.0 for the dashboard.

Is your Zitadel instance new? If so you might want to check out the Quick Start Guide. It comes with a one liner to set everything up (including Zitadel, Certificates, etc.). Before running the setup be sure that your domain is pointing to your server.

You can check the logs with docker compose logs dashboard, docker compose logs management

git-day commented 3 months ago

Thanks @heisbrot . Confirmed 2.5.0 is running.

The logs don't show much other than the following error related to cache. Could this be the problem with the dahsboard not logging in ?

management-1 | 2024-08-16T16:16:10Z WARN [context: SYSTEM] management/server/account.go:941: failed warming up cache due to error: unable to get zitadel token, statusCode 400

The dashboard simply never loads and loops with the following logo? What other logs should i look at? Should I look at enabling debug? image

git-day commented 3 months ago

@heisbrot at the client end using a browser, i see the following error message. Not sure where to take it from here. Any help is appreciated.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://nb.external.com:33073/api/users. (Reason: CORS request did not succeed). Status code: (null).

git-day commented 3 months ago

@heisbrot , I've made a few tweaks to the compose file, and am able to authenticate, much the same way as i previously had, but am now getting a Object { message: "token invalid", code: 401 } error through the browser.

Note really sure what to chase here, Netbird, Nginx or Zitadel. Based on the error above, appears to be Zitadel. Having said this, i have followed the official guides, and triple checked the settings. I have also searched through github for clues re the error, but noting firm that would resolve the issue.

Keen to hear what you think we should tackle here.

image

joshuahigginson1 commented 3 months ago

Don't want to lead you down the garden path @git-day , we had some CORS issues after hardening the Nginx instance which fronts the dashboard UI.

First off, I'd try launching a separate application and try to connect it to Zitadel directly.

I'd also try an a separate browser, in private mode, just to make sure that no expired auth tokens are being cached in local storage.

Is your url actually set to 'nb.external.com' or did you just sensor this when posting your issue?

git-day commented 3 months ago

@joshuahigginson1 , thanks for your help here. Let me try another app as you have suggest to auth against Zitadel. Just an FYI, i have run various browsers, Edge, Chrome, Firefox, Brave, including their own respective private modes.

Correct on the obfuscation re domain name.

I'm wondering why it's saying that it can't authenticate. sudo docker compose logs management shows a number of the following errors:

management-1 | 2024-08-17T08:57:42Z ERRO [context: HTTP, requestID: 0a964f71-3e3d-44e2-b510-76351aa50461] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get zitadel token, statusCode 400 management-1 | 2024-08-17T08:57:42Z ERRO [context: HTTP, requestID: 0a964f71-3e3d-44e2-b510-76351aa50461] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-08-17T08:57:42Z ERRO [context: HTTP, requestID: 0a964f71-3e3d-44e2-b510-76351aa50461] management/server/telemetry/http_api_metrics.go:191: HTTP response 0a964f71-3e3d-44e2-b510-76351aa50461: GET /api/users status 401

I'm considering trying another Idp, not sure which i should choose to do a quick test with.

git-day commented 3 months ago

@joshuahigginson1 and @heisbrot , i managed to run up an authentik Idp instance and reconfigured netbird to use it, and have resolved the issues encountered above. Zitadel does appear to have an issue, and is unresolved IMO.

In any instance, although i have cleared one hurdle, another one has popped up. This time its related to the an account within a cache. See error message below.

sudo docker compose logs management

management-1 | 2024-08-18T09:34:07Z INFO [context: HTTP, requestID: 8e78b404-1d03-4f6b-9e4f-82a00184879f, accountID: , userID: 4] management/server/account.go:1427: cache invalid. Users unknown to the cache: 2 management-1 | 2024-08-18T09:34:08Z INFO [requestID: 8e78b404-1d03-4f6b-9e4f-82a00184879f, accountID: , userID: 4, context: HTTP] management/server/account.go:1388: refreshing cache for account cqtkjnqde1ds738k3c00 management-1 | 2024-08-18T09:34:08Z WARN management/server/account.go:1261: user 280133991200194563 not found in IDP management-1 | 2024-08-18T09:34:08Z WARN [requestID: 8e78b404-1d03-4f6b-9e4f-82a00184879f, accountID: , userID: 4, context: HTTP] management/server/account.go:1395: cache for account cqtkjnqde1ds738k3c00 reached maximum refresh attempts (2) These issues appear to be related to the following open issue https://github.com/netbirdio/netbird/issues/1942

When I login through the frontend, it authenticates and redirects to the following page. The odd thing here is that i can't access the dashboard. Assuming this is because its related to the above user not found in IDP ?

How do i troubleshoot this?

image

joshuahigginson1 commented 3 months ago

Looks like Netbird has already managed to sync some users from your existing Zitadel setup, and now you've lost the ability to log in as an admin. If you can, this should be a matter of deleting your docker volumes and/or Postgres database. ☺️

git-day commented 3 months ago

@joshuahigginson1 thanks for coming back to me, really appreciate your support!

Yeah, I kind did that by blowing away the cloned github 'netbird' then I pulled down a the latest version, and figured the volumes required by the docker instances would have been removed, but this wasn't the case hence the info above. I've since built a new VM and started fresh, happy to report that the netbird dashboard has appeared and seems stable with Authentik as the Idp.

Having said this, keen to know if you would like me to open another issue re the challenges I had with Zitadel or continue in this thread. Happy to run up another instance for testing / resolution around netbird/Zitadel/Nginx. Let me know.

ShlomiPorush commented 3 months ago

netbird dashboard has appeared and seems stable with Authentik as the Idp

Hi, how did you manage to get Authentik to work with netbird. i'm stock on the same issues as you mentioned (https://github.com/netbirdio/netbird/issues/2226#issuecomment-2294799417) 2024-08-22T16:45:57Z WARN [context: SYSTEM] management/server/account.go:1022: failed warming up cache due to error: unable to get authentik token, statusCode 400

git-day commented 3 months ago

@ShlomiPorush we'll need to know a little more about your environment and setup. Assuming you are using a reverse proxy for both Netbird and Authentik?

The article i followed for Authentik setup is https://docs.netbird.io/selfhosted/identity-providers#authentik.

Note that my instance originally ran with Zitadel, which is the error that i received and have since removed Zitadel. When i stood up an instance of Authentik, i chose not to use a reverse proxy, and configured SSL natively with the Authentik instance. I did this to simply reduce any further troubleshooting on the Idp end. This allowed me to ensure that the Netbird config was correct.

Might be useful to look at the logs from the Netbird end sudo docker compose logs management and sudo docker compose logs dashboard.

bachrc commented 2 months ago

Same error here. Running a fresh install of netbird with Authentik, and it keeps disconnecting me on the dashboard

git-day commented 2 months ago

@bachrc we'll need to know a little more about your environment and setup. Assuming you are using a reverse proxy for both Netbird and Authentik?

bachrc commented 2 months ago

No reverse proxies for each, they are on separate vps and they handle themselves the TLS. Both are installed with their own docker

arshanskiyav commented 2 months ago

New install NetBird + KeyCloak (25.0.1) + NGINX (v14 on centos, grpc_socket_keepalive on - commented out, letsencrypt). Each service on a different virtual machine

TCP ports 443,80 forwarded to NGINX

I got:

layout-8d9e50216f3f6630.js:1 GET https://ntbd.domain.com:33073/api/users net::ERR_CONNECTION_TIMED_OUT

Then i add forwarding TCP ports 33073,33080 to NetBird and get this problem::

GET https://ntbd.domain.com:33073/api/users net::ERR_SSL_PROTOCOL_ERROR

git-day commented 2 months ago

@arshanskiyav can you please share the layout of the environment and where each service sits, as diag would be ideal. Is each service on a sperate network? Can you share the docker compose files for Netbird, NGNIX followed by the nginx.conf. It would also be useful to see the associated Netbird configs, management.json and. What idp are you using?

arshanskiyav commented 2 months ago

@arshanskiyav can you please share the layout of the environment and where each service sits, as diag would be ideal. Is each service on a sperate network? Can you share the docker compose files for Netbird, NGNIX followed by the nginx.conf. It would also be useful to see the associated Netbird configs, management.json and. What idp are you using?

Hi, thanks for your reply.

For idp i use - keycloak (v 25.0.1). Each service is located on a separate virtual machine, but on a shared network (10.110.1.0/24).

Nginx is standalone on Centos. Nginx version is 14. My nginx conf basis on the tmpl from netbird/infrastructure_files/nginx.tmpl.conf:

ntbd.domain.com.conf ``` upstream dashboard { # insert the http port of your dashboard container here # server vpn001.skz.loc:8011; server vpn001.skz.loc; # Improve performance by keeping some connections alive. keepalive 10; } upstream signal { # insert the grpc port of your signal container here server vpn001.skz.loc:10000; } upstream management { # insert the grpc+http port of your signal container here server vpn001.skz.loc:8012; } server { server_name ntbd.domain.com; # This is necessary so that grpc connections do not get closed early # see https://stackoverflow.com/a/67805465 client_header_timeout 1d; client_body_timeout 1d; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Scheme $scheme; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Host $host; grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Proxy dashboard location / { proxy_pass http://dashboard; } # Proxy Signal location /signalexchange.SignalExchange/ { grpc_pass grpc://signal; #grpc_ssl_verify off; grpc_read_timeout 1d; grpc_send_timeout 1d; # grpc_socket_keepalive on; } # Proxy Management http endpoint location /api { proxy_pass http://management; } # Proxy Management grpc endpoint location /management.ManagementService/ { grpc_pass grpc://management; #grpc_ssl_verify off; grpc_read_timeout 1d; grpc_send_timeout 1d; # grpc_socket_keepalive on; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/letsencrypt/live/ntbd.domain.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/ntbd.domain.com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = ntbd.domain.com) { return 301 https://$host$request_uri; } # managed by Certbot server_name ntbd.domain.com; listen 80; return 404; # managed by Certbot } ```
docker-compose.yml ``` version: "3" services: #UI dashboard dashboard: image: netbirdio/dashboard:latest restart: unless-stopped ports: - 80:80 - 443:443 environment: # Endpoints - NETBIRD_MGMT_API_ENDPOINT=https://ntbd.domain.com:33073 - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://ntbd.domain.com:33073 # OIDC - AUTH_AUDIENCE=netbird-client - AUTH_CLIENT_ID=netbird-client - AUTH_CLIENT_SECRET= - AUTH_AUTHORITY=https://sso.domain.com/realms/NETBIRD - USE_AUTH0=false - AUTH_SUPPORTED_SCOPES=openid profile email offline_access api - AUTH_REDIRECT_URI= - AUTH_SILENT_REDIRECT_URI= - NETBIRD_TOKEN_SOURCE=accessToken # SSL - NGINX_SSL_PORT=443 # Letsencrypt - LETSENCRYPT_DOMAIN= - LETSENCRYPT_EMAIL= volumes: - netbird-letsencrypt:/etc/letsencrypt/ logging: driver: "json-file" options: max-size: "500m" max-file: "2" # Signal signal: image: netbirdio/signal:latest restart: unless-stopped volumes: - netbird-signal:/var/lib/netbird ports: - 10000:80 # # port and command for Let's Encrypt validation # - 443:443 # command: ["--letsencrypt-domain", "", "--log-file", "console"] logging: driver: "json-file" options: max-size: "500m" max-file: "2" # Relay relay: image: netbirdio/relay:latest restart: unless-stopped environment: - NB_LOG_LEVEL=info - NB_LISTEN_ADDRESS=:33080 - NB_EXPOSED_ADDRESS=ntbd.domain.com:33080 # todo: change to a secure secret - NB_AUTH_SECRET=wxHQy5xhJP4 ports: - 33080:33080 logging: driver: "json-file" options: max-size: "500m" max-file: "2" # Management management: image: netbirdio/management:latest restart: unless-stopped depends_on: - dashboard volumes: - netbird-mgmt:/var/lib/netbird - netbird-letsencrypt:/etc/letsencrypt:ro - ./management.json:/etc/netbird/management.json ports: - 33073:443 #API port # # command for Let's Encrypt validation without dashboard container # command: ["--letsencrypt-domain", "", "--log-file", "console"] command: [ "--port", "443", "--log-file", "console", "--log-level", "info", "--disable-anonymous-metrics=false", "--single-account-mode-domain=ntbd.domain.com", "--dns-domain=netbird.selfhosted" ] logging: driver: "json-file" options: max-size: "500m" max-file: "2" environment: - NETBIRD_STORE_ENGINE_POSTGRES_DSN= # Coturn coturn: image: coturn/coturn:latest restart: unless-stopped #domainname: ntbd.domain.com # only needed when TLS is enabled volumes: - ./turnserver.conf:/etc/turnserver.conf:ro # - ./privkey.pem:/etc/coturn/private/privkey.pem:ro # - ./cert.pem:/etc/coturn/certs/cert.pem:ro network_mode: host command: - -c /etc/turnserver.conf logging: driver: "json-file" options: max-size: "500m" max-file: "2" volumes: netbird-mgmt: netbird-signal: netbird-letsencrypt: ```
management.json ``` { "Stuns": [ { "Proto": "udp", "URI": "stun:ntbd.domain.com:3478", "Username": "", "Password": "" } ], "TURNConfig": { "TimeBasedCredentials": false, "CredentialsTTL": "12h0m0s", "Secret": "secret", "Turns": [ { "Proto": "udp", "URI": "turn:ntbd.domain.com:3478", "Username": "self", "Password": "PjklExeusQlt6g" } ] }, "Relay": { "Addresses": [ "rel://ntbd.domain.com:33080" ], "CredentialsTTL": "24h0m0s", "Secret": "wxHQy5xhJP4" }, "Signal": { "Proto": "https", "URI": "ntbd.domain.com:10000", "Username": "", "Password": "" }, "Datadir": "/var/lib/netbird/", "DataStoreEncryptionKey": "p+CupLYbmWrg+Var+0EQ=", "HttpConfig": { "LetsEncryptDomain": "", "CertFile": "", "CertKey": "", "AuthAudience": "netbird-client", "AuthIssuer": "https://sso.domain.com/realms/NETBIRD", "AuthUserIDClaim": "", "AuthKeysLocation": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/certs", "OIDCConfigEndpoint": "https://sso.domain.com/realms/NETBIRD/.well-known/openid-configuration", "IdpSignKeyRefreshEnabled": false, "ExtraAuthAudience": "" }, "IdpManagerConfig": { "ManagerType": "keycloak", "ClientConfig": { "Issuer": "https://sso.domain.com/realms/NETBIRD", "TokenEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/token", "ClientID": "netbird-backend", "ClientSecret": "wd6xg2QMyo3", "GrantType": "client_credentials" }, "ExtraConfig": { "AdminEndpoint": "https://sso.domain.com/admin/realms/NETBIRD" }, "Auth0ClientCredentials": null, "AzureClientCredentials": null, "KeycloakClientCredentials": null, "ZitadelClientCredentials": null }, "DeviceAuthorizationFlow": { "Provider": "hosted", "ProviderConfig": { "ClientID": "netbird-client", "ClientSecret": "", "Domain": "sso.domain.com", "Audience": "netbird-client", "TokenEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/token", "DeviceAuthEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/auth/device", "AuthorizationEndpoint": "", "Scope": "openid", "UseIDToken": false, "RedirectURLs": null } }, "PKCEAuthorizationFlow": { "ProviderConfig": { "ClientID": "netbird-client", "ClientSecret": "", "Domain": "", "Audience": "netbird-client", "TokenEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/token", "DeviceAuthEndpoint": "", "AuthorizationEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/auth", "Scope": "openid profile email offline_access api", "UseIDToken": false, "RedirectURLs": [ "http://localhost:53000" ] } }, "StoreConfig": { "Engine": "sqlite" }, "ReverseProxy": { "TrustedHTTPProxies": [], "TrustedHTTPProxiesCount": 0, "TrustedPeers": [ "0.0.0.0/0" ] } } ```
diagram as i see it ![image](https://github.com/user-attachments/assets/9c1dd34a-6cbc-4324-96a8-2665a468268f)

I tried to change the external FQDN in /etc/hosts (ntbd.domain.com) resolution from external IP to 127.0.01 but I got the same error:

GET https://ntbd.domain.com:33073/api/users net::ERR_CONNECTION_TIMED_OUT

And I checked how the name is resolved on the Netbird virtual machine and in the Docker container "dashboard", the name is resolved as 127.0.0.1

Then i add forwarding TCP ports 33073,33080 to NetBird and get this problem::

GET https://ntbd.domain.com:33073/api/users net::ERR_SSL_PROTOCOL_ERROR

git-day commented 1 month ago

@arshanskiyav , try the following.

upstream dashboard {
    # insert the http port of your dashboard container here
#    server vpn001.skz.loc:8011;
    server 10.110.1.5:8086;

    # Improve performance by keeping some connections alive.
    keepalive 10;
}
upstream signal {
    # insert the grpc port of your signal container here
    server 10.110.1.5:8087;
}
upstream management {
    # insert the grpc+http port of your signal container here
    server 10.110.1.5:6443;
}
upstream relay {
    server 10.110.1.5:8088;
}

server {
        listen 443 ssl http2;
        server_name ntbd.domain.com;

    # This is necessary so that grpc connections do not get closed early
    # see https://stackoverflow.com/a/67805465
    client_header_timeout 1d;
    client_body_timeout 1d;

    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Scheme $scheme;
    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forwarded-Host $host;
    grpc_set_header         X-Forwarded-For $proxy_add_x_forwarded_for;

    # Proxy dashboard
    location / {
        proxy_pass http://dashboard;
    }
    # Proxy Signal
    location /signalexchange.SignalExchange/ {
        grpc_pass grpc://signal;
        #grpc_ssl_verify off;
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }
    # Proxy Management http endpoint
    location /api {
        proxy_pass http://management;
    }
    # Proxy Management grpc endpoint
    location /management.ManagementService/ {
        grpc_pass grpc://management;
        #grpc_ssl_verify off;
        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;
    }
    # Proxy Relay http endpoint
    location /relay/ {
        proxy_pass http://relay/relay;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";

        # Forward headers
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_cache_bypass $http_upgrade;
    }

    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/ntbd.domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/ntbd.domain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = ntbd.domain.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        server_name ntbd.domain.com;
        listen 80;
    return 404; # managed by Certbot
}
version: "3"
services:
  #UI dashboard
  dashboard:
    image: netbirdio/dashboard:latest
    restart: unless-stopped
    ports:
      - 8086:80
      #- 443:443
    environment:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://ntbd.domain.com:443
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://ntbd.domain.com:443
      # OIDC
      - AUTH_AUDIENCE=netbird-client
      - AUTH_CLIENT_ID=netbird-client
      - AUTH_CLIENT_SECRET=
      - AUTH_AUTHORITY=https://sso.domain.com/realms/NETBIRD
      - USE_AUTH0=false
      - AUTH_SUPPORTED_SCOPES=openid profile email offline_access api
      - AUTH_REDIRECT_URI=
      - AUTH_SILENT_REDIRECT_URI=
      - NETBIRD_TOKEN_SOURCE=accessToken
      # SSL
      - NGINX_SSL_PORT=443
      # Letsencrypt
      - LETSENCRYPT_DOMAIN=
      - LETSENCRYPT_EMAIL=
    volumes:
      - netbird-letsencrypt:/etc/letsencrypt/
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"
  # Signal
  signal:
    image: netbirdio/signal:latest
    restart: unless-stopped
    volumes:
      - netbird-signal:/var/lib/netbird
    ports:
      - 8087:80
  #      # port and command for Let's Encrypt validation
  #      - 443:443
  #    command: ["--letsencrypt-domain", "", "--log-file", "console"]
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"
  # Relay
  relay:
    image: netbirdio/relay:latest
    restart: unless-stopped
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:8088
    - NB_EXPOSED_ADDRESS=rels://ntbd.domain.com:443
    # todo: change to a secure secret
    - NB_AUTH_SECRET=wxHQy5xhJP4
    ports:
      - 8088:8088
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

  # Management
  management:
    image: netbirdio/management:latest
    restart: unless-stopped
    depends_on:
      - dashboard
    volumes:
      - netbird-mgmt:/var/lib/netbird
      - netbird-letsencrypt:/etc/letsencrypt:ro
      - ./management.json:/etc/netbird/management.json
    ports:
      - 6443:443 #API port
  #    # command for Let's Encrypt validation without dashboard container
  #    command: ["--letsencrypt-domain", "", "--log-file", "console"]
    command: [
      "--port", "443",
      "--log-file", "console",
      "--log-level", "info",
      "--disable-anonymous-metrics=false",
      "--single-account-mode-domain=ntbd.domain.com",
      "--dns-domain=netbird.selfhosted"
      ]
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"
    environment:
      - NETBIRD_STORE_ENGINE_POSTGRES_DSN=

  # Coturn
  coturn:
    image: coturn/coturn:latest
    restart: unless-stopped
    #domainname: ntbd.domain.com # only needed when TLS is enabled
    volumes:
      - ./turnserver.conf:/etc/turnserver.conf:ro
    #      - ./privkey.pem:/etc/coturn/private/privkey.pem:ro
    #      - ./cert.pem:/etc/coturn/certs/cert.pem:ro
    network_mode: host
    command:
      - -c /etc/turnserver.conf
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"
volumes:
  netbird-mgmt:
  netbird-signal:
  netbird-letsencrypt:
{
    "Stuns": [
        {
            "Proto": "udp",
            "URI": "stun:ntbd.domain.com:3478",
            "Username": "",
            "Password": ""
        }
    ],
    "TURNConfig": {
        "TimeBasedCredentials": false,
        "CredentialsTTL": "12h0m0s",
        "Secret": "secret",
        "Turns": [
            {
                "Proto": "udp",
                "URI": "turn:ntbd.domain.com:3478",
                "Username": "self",
                "Password": "PjklExeusQlt6g"
            }
        ]
    },
    "Relay": {
        "Addresses": [
            "rels://ntbd.domain.com:443"
        ],
        "CredentialsTTL": "24h0m0s",
        "Secret": "wxHQy5xhJP4"
    },
    "Signal": {
        "Proto": "https",
        "URI": "ntbd.domain.com:443",
        "Username": "",
        "Password": ""
    },
    "Datadir": "/var/lib/netbird/",
    "DataStoreEncryptionKey": "p+CupLYbmWrg+Var+0EQ=",
    "HttpConfig": {
        "LetsEncryptDomain": "",
        "CertFile": "",
        "CertKey": "",
        "AuthAudience": "netbird-client",
        "AuthIssuer": "https://sso.domain.com/realms/NETBIRD",
        "AuthUserIDClaim": "",
        "AuthKeysLocation": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/certs",
        "OIDCConfigEndpoint": "https://sso.domain.com/realms/NETBIRD/.well-known/openid-configuration",
        "IdpSignKeyRefreshEnabled": false,
        "ExtraAuthAudience": ""
    },
    "IdpManagerConfig": {
        "ManagerType": "keycloak",
        "ClientConfig": {
            "Issuer": "https://sso.domain.com/realms/NETBIRD",
            "TokenEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/token",
            "ClientID": "netbird-backend",
            "ClientSecret": "wd6xg2QMyo3",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "AdminEndpoint": "https://sso.domain.com/admin/realms/NETBIRD"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },
    "DeviceAuthorizationFlow": {
        "Provider": "hosted",
        "ProviderConfig": {
            "ClientID": "netbird-client",
            "ClientSecret": "",
            "Domain": "sso.domain.com",
            "Audience": "netbird-client",
            "TokenEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/token",
            "DeviceAuthEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/auth/device",
            "AuthorizationEndpoint": "",
            "Scope": "openid",
            "UseIDToken": false,
            "RedirectURLs": null
        }
    },
    "PKCEAuthorizationFlow": {
        "ProviderConfig": {
            "ClientID": "netbird-client",
            "ClientSecret": "",
            "Domain": "",
            "Audience": "netbird-client",
            "TokenEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/token",
            "DeviceAuthEndpoint": "",
            "AuthorizationEndpoint": "https://sso.domain.com/realms/NETBIRD/protocol/openid-connect/auth",
            "Scope": "openid profile email offline_access api",
            "UseIDToken": false,
            "RedirectURLs": [
                "http://localhost:53000"
            ]
        }
    },
    "StoreConfig": {
        "Engine": "sqlite"
    },
    "ReverseProxy": {
        "TrustedHTTPProxies": [],
        "TrustedHTTPProxiesCount": 0,
        "TrustedPeers": [
            "0.0.0.0/0"
        ]
    }
}
git-day commented 4 days ago

All, adding in the authentik management.json config.

management.json - authentik

{
    "Stuns": [
        {
            "Proto": "udp",
            "URI": "stun:ntbd.domain.com:3478",
            "Username": "",
            "Password": ""
        }
    ],
    "TURNConfig": {
        "TimeBasedCredentials": false,
        "CredentialsTTL": "12h0m0s",
        "Secret": "secret",
        "Turns": [
            {
                "Proto": "udp",
                "URI": "turn:ntbd.domain.com:3478",
                "Username": "self",
                "Password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
            }
        ]
    },
    "Signal": {
        "Proto": "https",
        "URI": "ntbd.domain.com",
        "Username": "",
        "Password": ""
    },
    "Datadir": "/var/lib/netbird/",
    "DataStoreEncryptionKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "HttpConfig": {
        "LetsEncryptDomain": "",
        "CertFile": "",
        "CertKey": "",
        "AuthAudience": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "AuthIssuer": "https://sso.domain.com/application/o/netbird/",
        "AuthUserIDClaim": "",
        "AuthKeysLocation": "https://sso.domain.com/application/o/netbird/jwks/",
        "OIDCConfigEndpoint": "https://sso.domain.com/application/o/netbird/.well-known/openid-configuration",
        "IdpSignKeyRefreshEnabled": false,
        "ExtraAuthAudience": ""
    },
    "IdpManagerConfig": {
        "ManagerType": "authentik",
        "ClientConfig": {
            "Issuer": "https://sso.domain.com/application/o/netbird",
            "TokenEndpoint": "https://sso.domain.com/application/o/token/",
            "ClientID": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
            "ClientSecret": "",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "Password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
            "Username": "ENTER USERNAME"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },
    "DeviceAuthorizationFlow": {
        "Provider": "hosted",
        "ProviderConfig": {
            "ClientID": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
            "ClientSecret": "",
            "Domain": "sso.domain.com",
            "Audience": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
            "TokenEndpoint": "https://sso.domain.com/application/o/token/",
            "DeviceAuthEndpoint": "https://sso.domain.com/application/o/device/",
            "AuthorizationEndpoint": "",
            "Scope": "openid",
            "UseIDToken": false,
            "RedirectURLs": null
        }
    },
    "PKCEAuthorizationFlow": {
        "ProviderConfig": {
            "ClientID": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
            "ClientSecret": "",
            "Domain": "",
            "Audience": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
            "TokenEndpoint": "https://sso.domain.com/application/o/token/",
            "DeviceAuthEndpoint": "",
            "AuthorizationEndpoint": "https://sso.domain.com/application/o/authorize/",
            "Scope": "openid profile email offline_access api",
            "UseIDToken": false,
            "RedirectURLs": [
                "http://localhost:53000"
            ]
        }
    },
    "StoreConfig": {
      "Engine": "sqlite"
  },
  "Relay": {
      "Addresses": ["rels://ntbd.domain.com/relay"],
      "CredentialsTTL": "24h",
      "Secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  },
  "ReverseProxy": {
      "TrustedHTTPProxies": [],
      "TrustedHTTPProxiesCount": 0,
      "TrustedPeers": [
          "0.0.0.0/0"
      ]
  }
}