luckylinux commented 6 months ago

Describe the problem

I am facing lots of different issues during the setup of Netbird and Authentik (self hosted).

Following the tutorial is quite straightforward and most of the things seem to work.

I am stuck in the part that relates to the "netbird-management" container.

I use Cloudflare Proxy DNS and I had "Your SSL/TLS encryption mode" set to "Off". Now I turned it to "Full (Strict)" based on some guidelines on Cloudflare.

I also added 127.0.0.1 auth.MYDOMAIN.TLD to /etc/hosts (on the Podman Host) but that did not really solve the issues.

There are just too many redirects / connection refused / sometimes invalid X509 certificate (the latter probably when using the /etc/hosts entry line) going on here.

2024-03-09T19:39:09Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration Get "https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration": stopped after 10 redirects
2024-03-09T19:41:37Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:41:38Z INFO management/cmd/management.go:452: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:41:38Z INFO management/cmd/management.go:454: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:458: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:463: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:466: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:484: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:487: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: 
2024-03-09T19:41:38Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-09T19:41:38Z INFO management/server/store.go:94: using SQLite store engine
Error: failed retrieving a new idp manager with err: authentik IdP configuration is incomplete, TokenEndpoint is missing
2024-03-09T19:45:00Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:45:01Z INFO management/cmd/management.go:452: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:45:01Z INFO management/cmd/management.go:454: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:458: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:463: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:466: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:484: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:487: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: 
2024-03-09T19:45:01Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-09T19:45:01Z INFO management/server/store.go:94: using SQLite store engine
Error: failed retrieving a new idp manager with err: authentik IdP configuration is incomplete, TokenEndpoint is missing

To Reproduce

Steps to reproduce the behavior:

compose.yml file


version: "3"
services:
#UI dashboard
dashboard:
image: wiretrustee/dashboard:latest
restart: unless-stopped
container_name: netbird-dashboard
#ports:
#  - 80:80
#  - 443:443
networks:
  - traefik
environment:
  # Endpoints
  - NETBIRD_MGMT_API_ENDPOINT=https://netbird.MYDOMAIN.TLD:33073
  - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.MYDOMAIN.TLD:33073
  # OIDC
  - AUTH_AUDIENCE=<CLIENT_ID>
  - AUTH_CLIENT_ID=<CLIENT_ID>
  - AUTH_CLIENT_SECRET=
  - AUTH_AUTHORITY=
  - USE_AUTH0=false
  - AUTH_SUPPORTED_SCOPES=openid profile email offline_access api
  - AUTH_REDIRECT_URI=
  - AUTH_SILENT_REDIRECT_URI=
  - NETBIRD_TOKEN_SOURCE=accessToken
  # SSL
  #- NGINX_SSL_PORT=443
  # Letsencrypt
  #- LETSENCRYPT_DOMAIN=
  #- LETSENCRYPT_EMAIL=myemail@DOMAIN.TLD
#    volumes:
#      - netbird-letsencrypt:/etc/letsencrypt/
labels:
- traefik.enable=true
- traefik.http.routers.netbird-dashboard.rule=Host(`netbird.MYDOMAIN.TLD`)
- traefik.http.services.netbird-dashboard.loadbalancer.server.port=80

# Signal
signal:
image: netbirdio/signal:latest
container_name: netbird-signal
restart: unless-stopped
volumes:
  #- netbird-signal:/var/lib/netbird
  - ~/containers/data/netbird/signal:/var/lib/netbird
networks:
  - traefik
#ports:
#  - 10000:80
#      # port and command for Let's Encrypt validation
#      - 443:443
#    command: ["--letsencrypt-domain", "", "--log-file", "console"]
labels:
- traefik.enable=true
- traefik.http.routers.netbird-signal.rule=Host(`netbird.MYDOMAIN.TLD`) && PathPrefix(`/signalexchange.SignalExchange/`)
- traefik.http.services.netbird-signal.loadbalancer.server.port=80
- traefik.http.services.netbird-signal.loadbalancer.server.scheme=h2c

# Management
management:
image: netbirdio/management:latest
#    restart: unless-stopped
#    restart: no
container_name: netbird-management
depends_on:
  - dashboard
volumes:
#      - netbird-mgmt:/var/lib/netbird
  - ~/containers/data/netbird/management:/var/lib/netbird
#      - netbird-letsencrypt:/etc/letsencrypt:ro
  - ~/containers/config/netbird/management.json:/etc/netbird/management.json
networks:
  - traefik
#ports:
#  - 33073:443 #API port
#    # command for Let's Encrypt validation without dashboard container
#    command: ["--letsencrypt-domain", "", "--log-file", "console"]
command: [
  "--port", "80",
  "--log-file", "console",
  "--disable-anonymous-metrics=true",
  "--single-account-mode-domain=netbird.MYDOMAIN.TLD",
  "--dns-domain=MYDOMAIN.TLD"
  ]
labels:
- traefik.enable=true
- traefik.http.routers.netbird-api.rule=Host(`netbird.MYDOMAIN.TLD`) && PathPrefix(`/api`)
- traefik.http.routers.netbird-api.service=netbird-api
- traefik.http.services.netbird-api.loadbalancer.server.port=80

- traefik.http.routers.netbird-management.rule=Host(`netbird.MYDOMAIN.TLD`) && PathPrefix(`/management.ManagementService/`)
- traefik.http.routers.netbird-management.service=netbird-management
- traefik.http.services.netbird-management.loadbalancer.server.port=80
- traefik.http.services.netbird-management.loadbalancer.server.scheme=h2c

# Coturn
coturn:
image: coturn/coturn:latest
container_name: netbird-coturn
restart: unless-stopped
domainname: netbird.MYDOMAIN.TLD
volumes:
  - ~/containers/config/netbird/turnserver.conf:/etc/turnserver.conf:ro
#      - ./privkey.pem:/etc/coturn/private/privkey.pem:ro
#      - ./cert.pem:/etc/coturn/certs/cert.pem:ro
networks:
  - traefik
network_mode: host
command:
  - -c /etc/turnserver.conf

volumes:

netbird-mgmt:

netbird-signal:

netbird-letsencrypt:

networks: traefik: external: true


3. Run `podman compose up -d`
4. See that `netbird-management` is not running when issueing `podman ps`'
5. Run `podman logs netbird-management`

**Expected behavior**

The container `netbird-management` to start successfully. Not being stuck in a redirect / too many redirects loops causing the `netbird-management` container to automatically restart, `traefik` attemping a certificate renewal and ending up banned from Letsencrypt for a few hours.

**Are you using NetBird Cloud?**

Self-Hosted Control Panel.

**NetBird version**

Cannot enter management console since it keeps crashing  / restarting ...
Was installed & pulled from Docker Hub today.

**NetBird status -d output:**

Cannot enter management console since it keeps crashing  / restarting ...
Was installed & pulled from Docker Hub today.

**Screenshots**

N/A

**Additional context**

Running `podman` instead of `docker`.

Installed on Debian AMD64 12 Bookworm with APT Pinning of Podman (and conmon, podman-compose, ...) from Trixie/Testing.

`podman info` shows

host: arch: amd64 buildahVersion: 1.33.5 cgroupControllers:

cpu
memory
pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon_2.1.10+ds1-1_amd64 path: /usr/bin/conmon version: 'conmon version 2.1.10, commit: unknown' cpuUtilization: idlePercent: 91.15 systemPercent: 2.63 userPercent: 6.22 cpus: 1 databaseBackend: sqlite distribution: codename: bookworm distribution: debian version: "12" eventLogger: journald freeLocks: 2034 hostname: ra idMappings: gidmap:
- container_id: 0 host_id: 1001 size: 1
- container_id: 1 host_id: 165536 size: 65536 uidmap:
- container_id: 0 host_id: 1001 size: 1
- container_id: 1 host_id: 165536 size: 65536 kernel: 6.1.0-18-amd64 linkmode: dynamic logDriver: journald memFree: 112033792 memTotal: 2012446720 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: aardvark-dns_1.4.0-5_amd64 path: /usr/lib/podman/aardvark-dns version: aardvark-dns 1.4.0 package: netavark_1.4.0-3_amd64 path: /usr/lib/podman/netavark version: netavark 1.4.0 ociRuntime: name: crun package: crun_1.14.1-1_amd64 path: /usr/bin/crun version: |- crun version 1.14.1 commit: de537a7965bfbe9992e2cfae0baeb56a08128171 rundir: /run/user/1001/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL os: linux pasta: executable: /usr/bin/pasta package: passt_0.0~git20230309.7c7625d-1_amd64 version: | pasta unknown version Copyright Red Hat GNU Affero GPL version 3 or later https://www.gnu.org/licenses/agpl-3.0.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: exists: true path: /run/user/1001/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns_1.2.0-1_amd64 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.4 swapFree: 604762112 swapTotal: 1023406080 uptime: 320h 15m 8.00s (Approximately 13.33 days) variant: "" plugins: authorization: null log:
k8s-file
none
passthrough
journald network:
bridge
macvlan
ipvlan volume:
local registries: search:
registry.fedoraproject.org
registry.access.redhat.com
docker.io
quay.io store: configFile: /home/podman/.config/containers/storage.conf containerStore: number: 9 paused: 0 running: 8 stopped: 1 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs_1.13-1_amd64 Version: |- fusermount3 version: 3.14.0 fuse-overlayfs: version 1.13-dev FUSE library version 3.14.0 using FUSE kernel interface version 7.31 overlay.mountopt: nodev,metacopy=on graphRoot: /home/podman/containers/storage graphRootAllocated: 18969468928 graphRootUsed: 7787802624 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Supports shifting: "true" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 12 runRoot: /run/user/1001 transientStore: false volumePath: /home/podman/containers/storage/volumes version: APIVersion: 4.9.3 Built: 0 BuiltTime: Thu Jan 1 01:00:00 1970 GitCommit: "" GoVersion: go1.21.6 Os: linux OsArch: linux/amd64 Version: 4.9.3

luckylinux commented 6 months ago

OK I resolved the missing TokenEndpoint by putting:

"TokenEndpoint": "https://auth.MYDOMAIN.TLD/application/o/netbird/oauth/v2/token"

In management.json

(based on the findings from https://github.com/netbirdio/netbird/issues/750)

But now I get a different error

2024-03-09T19:39:09Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration Get "https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration": stopped after 10 redirects
2024-03-09T19:41:37Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:41:38Z INFO management/cmd/management.go:452: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:41:38Z INFO management/cmd/management.go:454: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:458: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:463: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:466: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:484: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:41:38Z INFO management/cmd/management.go:487: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: 
2024-03-09T19:41:38Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-09T19:41:38Z INFO management/server/store.go:94: using SQLite store engine
Error: failed retrieving a new idp manager with err: authentik IdP configuration is incomplete, TokenEndpoint is missing
2024-03-09T19:45:00Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:45:01Z INFO management/cmd/management.go:452: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-09T19:45:01Z INFO management/cmd/management.go:454: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:458: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:463: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:466: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:484: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: 
2024-03-09T19:45:01Z INFO management/cmd/management.go:487: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: 
2024-03-09T19:45:01Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-09T19:45:01Z INFO management/server/store.go:94: using SQLite store engine
Error: failed retrieving a new idp manager with err: authentik IdP configuration is incomplete, TokenEndpoint is missing
2024-03-10T08:47:40Z INFO management/cmd/management.go:447: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-10T08:47:41Z INFO management/cmd/management.go:452: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-10T08:47:41Z INFO management/cmd/management.go:454: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: 
2024-03-10T08:47:41Z INFO management/cmd/management.go:458: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: 
2024-03-10T08:47:41Z INFO management/cmd/management.go:463: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/oauth/v2/token
2024-03-10T08:47:41Z INFO management/cmd/management.go:466: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: 
2024-03-10T08:47:41Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: 
2024-03-10T08:47:41Z INFO management/cmd/management.go:484: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/oauth/v2/token
2024-03-10T08:47:41Z INFO management/cmd/management.go:487: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: 
2024-03-10T08:47:41Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-10T08:47:41Z INFO management/server/store.go:94: using SQLite store engine
2024-03-10T08:47:41Z INFO management/cmd/management.go:159: update config with activity store key
2024-03-10T08:48:12Z INFO management/cmd/management.go:171: geo location service has been initialized from /var/lib/netbird/
2024-03-10T08:48:12Z INFO management/server/account.go:849: single account mode enabled, accounts number 0
2024-03-10T08:48:12Z WARN management/cmd/management.go:185: TrustedPeers are configured to default value '0.0.0.0/0', '::/0'. This allows connection IP spoofing.
2024-03-10T08:48:14Z WARN management/server/account.go:889: failed warming up cache due to error: Post "https://auth.MYDOMAIN.TLD/application/o/netbird/oauth/v2/token": EOF
Error: failed creating JWT validator: Get "https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/": EOF

luckylinux commented 6 months ago

Any update ? Or any idea what log file I should provide and how ?

mlsmaycon commented 6 months ago

@luckylinux, as stated in #1715, you are using Keycloak now. Do you want to continue troubleshooting this issue? If so, it seems like the IDP Manager configuration is incomplete; you can share its values by masking domains and secrets.

It is part of the management.json, with the key IdpManagerConfig

luckylinux commented 6 months ago

@mlsmaycon Well for now I'm not really using anything as nothing is working, for different reasons:

Authentik: the issue described here
Keycloak: unauthenticated
Zitadel: couldn't even try, the Zitadel container itself was causing I/O timeout issues probably due to very high CPU usage

So, for now, we can keep going with Keycloak (I don't really care, I just would like netbird+something+traefik+homeassistant to be able to access my home assistant instance remotely).

But ultimately it should also work with Authentik IMHO ...

luckylinux commented 6 months ago

I masked 2 values and replaced them by AUTHENTIK_CLIENT_ID_SECRET and AUTHENTIK_CLIENT_PASSWORD_SECRET

    "IdpManagerConfig": {
        "ManagerType": "authentik",
        "ClientConfig": {
            "Issuer": "",
            "TokenEndpoint": "https://auth.MYDOMAIN.TLD/application/o/netbird/oauth/v2/token",
            "ClientID": "AUTHENTIK_CLIENT_ID_SECRET",
            "ClientSecret": "",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "Password": "AUTHENTIK_CLIENT_PASSWORD_SECRET",
            "Username": "netbird"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    }

mlsmaycon commented 6 months ago

Thanks for sharing it @luckylinux , it seems the Issuer is also missing. These values get filled via the configure.sh script, if you still have your setup.env file, you can run it again and ensure that the TokenEndpoint and Issuer are set, as well as the client ID, Password, and Usernames.

As for the Username, the guide creates a Netbird user with uppercase N. Can you double-check that, too?

As an alternative, to get started, you can disable the IdpManagerConfig by setting the ManagerType to "none." that won't map user IDs to email addresses but should allow you to explore the platform.

luckylinux commented 6 months ago

Thanks for sharing it @luckylinux , it seems the Issuer is also missing. These values get filled via the configure.sh script, if you still have your setup.env file, you can run it again and ensure that the TokenEndpoint and Issuer are set, as well as the client ID, Password, and Usernames.

As for the Username, the guide creates a Netbird user with uppercase N. Can you double-check that, too?

As an alternative, to get started, you can disable the IdpManagerConfig by setting the ManagerType to "none." that won't map user IDs to email addresses but should allow you to explore the platform.

Netbird lowercase is intentional. My user is netbird :).

What should I set for issuer ? I do not have such a parameter in setup.env. How should I set it and to what ?

mlsmaycon commented 6 months ago

You can run:

apt install jq
curl https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration  | jq

That will give you JSON response with the issuer value.

luckylinux commented 6 months ago

OK ... That would be

issuer: "https://auth.MYDOMAIN.TLD/application/o/netbird/"

(I just grabbed it from Firefox :))

luckylinux commented 6 months ago

I fixed that in management.json by setting the issuer in IdpManagerConfig but the container netbird-management continuously crashes.

The logs show

2024-03-19T06:06:19Z INFO management/cmd/management.go:449: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration Get "https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration": EOF

Very similar / same error as happened before.

luckylinux commented 6 months ago

Any update ? Google didn't return similar error messages ...

I could validate both using jq empty $file.json or cat $file.json | jq empty both for

/etc/netbird/management.json:
Contents of https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration (saved using Firefox)

Removed auth.MYDOMAIN.TLD from /etc/hosts to try if it gave different results.

I could download locally (from the host serving the containers, i.e. remotely): curl https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration

Restarted all netbird containers.

Now I am getting:

2024-03-26T19:08:34Z INFO management/cmd/management.go:449: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-26T19:08:37Z INFO management/cmd/management.go:454: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-26T19:08:37Z INFO management/cmd/management.go:456: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/
2024-03-26T19:08:37Z INFO management/cmd/management.go:460: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/
2024-03-26T19:08:37Z INFO management/cmd/management.go:465: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/token/
2024-03-26T19:08:37Z INFO management/cmd/management.go:468: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/device/
2024-03-26T19:08:37Z INFO management/cmd/management.go:476: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: auth.MYDOMAIN.TLD
2024-03-26T19:08:37Z INFO management/cmd/management.go:486: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/token/
2024-03-26T19:08:37Z INFO management/cmd/management.go:489: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/authorize/
2024-03-26T19:08:37Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-26T19:08:37Z INFO management/server/store.go:92: using SQLite store engine
2024-03-26T19:08:39Z INFO management/cmd/management.go:172: geo location service has been initialized from /var/lib/netbird/
2024-03-26T19:08:39Z INFO management/server/account.go:848: single account mode enabled, accounts number 0
2024-03-26T19:08:39Z WARN management/cmd/management.go:186: TrustedPeers are configured to default value '0.0.0.0/0', '::/0'. This allows connection IP spoofing.
Error: failed creating JWT validator: invalid character 'e' looking for beginning of value

It would be nice if the line number of the error would also be reported ...

Any idea ?

luckylinux commented 6 months ago

Now actually it's going a bit further, netbird-management does not crash, but I still cannot see a webpage working...

2024-03-26T19:08:34Z INFO management/cmd/management.go:449: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-26T19:08:37Z INFO management/cmd/management.go:454: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-26T19:08:37Z INFO management/cmd/management.go:456: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/
2024-03-26T19:08:37Z INFO management/cmd/management.go:460: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/
2024-03-26T19:08:37Z INFO management/cmd/management.go:465: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/token/
2024-03-26T19:08:37Z INFO management/cmd/management.go:468: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/device/
2024-03-26T19:08:37Z INFO management/cmd/management.go:476: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: auth.MYDOMAIN.TLD
2024-03-26T19:08:37Z INFO management/cmd/management.go:486: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/token/
2024-03-26T19:08:37Z INFO management/cmd/management.go:489: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/authorize/
2024-03-26T19:08:37Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-26T19:08:37Z INFO management/server/store.go:92: using SQLite store engine
2024-03-26T19:08:39Z INFO management/cmd/management.go:172: geo location service has been initialized from /var/lib/netbird/
2024-03-26T19:08:39Z INFO management/server/account.go:848: single account mode enabled, accounts number 0
2024-03-26T19:08:39Z WARN management/cmd/management.go:186: TrustedPeers are configured to default value '0.0.0.0/0', '::/0'. This allows connection IP spoofing.
Error: failed creating JWT validator: invalid character 'e' looking for beginning of value
2024-03-26T19:11:55Z INFO management/cmd/management.go:449: loading OIDC configuration from the provided IDP configuration endpoint https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-26T19:11:56Z INFO management/cmd/management.go:454: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.MYDOMAIN.TLD/application/o/netbird/.well-known/openid-configuration
2024-03-26T19:11:56Z INFO management/cmd/management.go:456: overriding HttpConfig.AuthIssuer with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/
2024-03-26T19:11:56Z INFO management/cmd/management.go:460: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/netbird/jwks/
2024-03-26T19:11:56Z INFO management/cmd/management.go:465: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/token/
2024-03-26T19:11:56Z INFO management/cmd/management.go:468: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/device/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/device/
2024-03-26T19:11:56Z INFO management/cmd/management.go:476: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.MYDOMAIN.TLD, previously configured value: auth.MYDOMAIN.TLD
2024-03-26T19:11:56Z INFO management/cmd/management.go:486: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/token/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/token/
2024-03-26T19:11:56Z INFO management/cmd/management.go:489: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.MYDOMAIN.TLD/application/o/authorize/, previously configured value: https://auth.MYDOMAIN.TLD/application/o/authorize/
2024-03-26T19:11:56Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-03-26T19:11:56Z INFO management/server/store.go:92: using SQLite store engine
2024-03-26T19:11:57Z INFO management/cmd/management.go:172: geo location service has been initialized from /var/lib/netbird/
2024-03-26T19:11:57Z INFO management/server/account.go:848: single account mode enabled, accounts number 0
2024-03-26T19:11:57Z WARN management/cmd/management.go:186: TrustedPeers are configured to default value '0.0.0.0/0', '::/0'. This allows connection IP spoofing.
2024-03-26T19:11:58Z WARN management/server/account.go:888: failed warming up cache due to error: unable to get authentik token, statusCode 405
2024-03-26T19:11:59Z INFO management/cmd/management.go:287: running gRPC backward compatibility server: [::]:33073
2024-03-26T19:11:59Z INFO management/cmd/management.go:319: management server version 0.26.3
2024-03-26T19:11:59Z INFO management/cmd/management.go:320: running HTTP server and gRPC server on the same port: [::]:80

luckylinux commented 2 weeks ago

@mlsmaycon: any News on this ? I just tried again with an Updated v0.29.3 Install now, but same Error.

Where is this originating from ?

[netbird-management] | Error: failed creating JWT validator: invalid character 'e' looking for beginning of value

That seems to be the only Error that I can see at least ... Possibly there is some other Stuff going on with Traefik Reverse Proxy and Cloudflare DNS Proxy ???

netbirdio / netbird

Self Hosted Netbird + Authentik - Error: failed retrieving a new idp manager with err: authentik IdP configuration is incomplete, TokenEndpoint is missing - Error: failed creating JWT validator: Get ... EOF #1684

volumes:

netbird-mgmt:

netbird-signal:

netbird-letsencrypt: