Request failed with status code 401. Error: token invalid

[Unreeling8562]

Describe the problem

I installed Netbird with keycloak as idp. When I try to login I get the following error: Request failed with status code 401. Error: token invalid

In docker logs:

management-1  | 2024-06-16T15:25:56Z ERRO management/server/http/middleware/auth_middleware.go:88: Error when validating JWT claims: Error parsing token: invalid audience
management-1  | 2024-06-16T15:25:56Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
management-1  | 2024-06-16T15:25:56Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2776902179: GET /api/users status 401

To Reproduce

Steps to reproduce the behavior:

1. Go to Netbird
2. Try to login using Keycloak
3. Get error message

Expected behavior

I should see the dashboard after logging in

Are you using NetBird Cloud?

No, selfhosted on Hetzner X22

NetBird version

0.27.10

Screenshots

Additional context

Here is my setup.env:

## example file, you can copy this file to setup.env and update its values
##

# Image tags
# you can force specific tags for each component; will be set to latest if empty
NETBIRD_DASHBOARD_TAG=""
NETBIRD_SIGNAL_TAG=""
NETBIRD_MANAGEMENT_TAG=""
COTURN_TAG=""

# Dashboard domain. e.g. app.mydomain.com
NETBIRD_DOMAIN="netbird.example.com"

# TURN server domain. e.g. turn.mydomain.com
# if not specified it will assume NETBIRD_DOMAIN
NETBIRD_TURN_DOMAIN=""

# TURN server public IP address
# required for a connection involving peers in
# the same network as the server and external peers
# usually matches the IP for the domain set in NETBIRD_TURN_DOMAIN
NETBIRD_TURN_EXTERNAL_IP="xx.xx.xxx.xx"

# -------------------------------------------
# OIDC
#  e.g., https://example.eu.auth0.com/.well-known/openid-configuration
# -------------------------------------------
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://auth.example.com/realms/myrealm/.well-known/openid-configuration"
# The default setting is to transmit the audience to the IDP during authorization. However,
# if your IDP does not have this capability, you can turn this off by setting it to false.
#NETBIRD_DASH_AUTH_USE_AUDIENCE=false
NETBIRD_AUTH_AUDIENCE="netbird-client"
# e.g. netbird-client
NETBIRD_AUTH_CLIENT_ID="netbird-client"
# indicates the scopes that will be requested to the IDP
NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"
# NETBIRD_AUTH_CLIENT_SECRET is required only by Google workspace.
# NETBIRD_AUTH_CLIENT_SECRET=""
# if you want to use a custom claim for the user ID instead of 'sub', set it here
# NETBIRD_AUTH_USER_ID_CLAIM=""
# indicates whether to use Auth0 or not: true or false
NETBIRD_USE_AUTH0="false"
# if your IDP provider doesn't support fragmented URIs, configure custom
# redirect and silent redirect URIs, these will be concatenated into your NETBIRD_DOMAIN domain.
# NETBIRD_AUTH_REDIRECT_URI="/peers"
# NETBIRD_AUTH_SILENT_REDIRECT_URI="/add-peers"
# Updates the preference to use id tokens instead of access token on dashboard
# Okta and Gitlab IDPs can benefit from this
# NETBIRD_TOKEN_SOURCE="idToken"
# -------------------------------------------
# OIDC Device Authorization Flow
# -------------------------------------------
NETBIRD_AUTH_DEVICE_AUTH_PROVIDER="none"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID=""
# Some IDPs requires different audience, scopes and to use id token for device authorization flow
# you can customize here:
NETBIRD_AUTH_DEVICE_AUTH_AUDIENCE=$NETBIRD_AUTH_AUDIENCE
NETBIRD_AUTH_DEVICE_AUTH_SCOPE="openid"
NETBIRD_AUTH_DEVICE_AUTH_USE_ID_TOKEN=false
# -------------------------------------------
# OIDC PKCE Authorization Flow
# -------------------------------------------
# Comma separated port numbers. if already in use, PKCE flow will choose an available port from the list as an alternative
# eg. 53000,54000
NETBIRD_AUTH_PKCE_REDIRECT_URL_PORTS="53000"
# -------------------------------------------
# IDP Management
# -------------------------------------------
# eg. zitadel, auth0, azure, keycloak
NETBIRD_MGMT_IDP="keycloak"
# Some IDPs requires different client id and client secret for management api
NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend"
NETBIRD_IDP_MGMT_CLIENT_SECRET="MySuperSecureClientSecret"
# Required when setting up with Keycloak "https://<YOUR_KEYCLOAK_HOST_AND_PORT>/admin/realms/netbird"
NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://auth.example.com/admin/realms/myrealm"
# With some IDPs may be needed enabling automatic refresh of signing keys on expire
# NETBIRD_MGMT_IDP_SIGNKEY_REFRESH=false
# NETBIRD_IDP_MGMT_EXTRA_ variables. See https://docs.netbird.io/selfhosted/identity-providers for more information about your IDP of choice.
# -------------------------------------------
# Letsencrypt
# -------------------------------------------
# Disable letsencrypt
#  if disabled, cannot use HTTPS anymore and requires setting up a reverse-proxy to do it instead
NETBIRD_DISABLE_LETSENCRYPT=false
# e.g. hello@mydomain.com
NETBIRD_LETSENCRYPT_EMAIL="mail@example.com"
# -------------------------------------------
# Extra settings
# -------------------------------------------
# Disable anonymous metrics collection, see more information at https://netbird.io/docs/FAQ/metrics-collection
NETBIRD_DISABLE_ANONYMOUS_METRICS=true
# DNS DOMAIN configures the domain name used for peer resolution. By default it is netbird.selfhosted
NETBIRD_MGMT_DNS_DOMAIN=netbird.selfhosted

[Unreeling8562]

I've followed the official advanced docs from Netbird

[Cikaros]

It requires the Geo database to be installed. Check whether the database is installed. https://docs.netbird.io/selfhosted/geo-support

[Unreeling8562]

It requires the Geo database to be installed. Check whether the database is installed. https://docs.netbird.io/selfhosted/geo-support

I've installed this, but still the same error unfortunately

[landmass-deftly-reptile-budget]

I can imagine this is the same issue like this one (except this one is about Zitadel as an IDP): https://github.com/netbirdio/netbird/pull/2089

I doubt a 401 token invalid error has something to do with the geo database.

[Cikaros]

Check the logs of the management service

[HansAschauer]

I had a problem with quite similar symptoms. This is possibly a bug in recent versions of keycloak, but I am not an expert with it.

However, I could work around the issue in the following way:

In the setup guide (https://docs.netbird.io/selfhosted/identity-providers#step-6-create-a-net-bird-client-scope), go to step 6 ("Create a NetBird client scope"). But instead of adding "netbird-client" to "Included Client Audience", add it to "Included Custom Audience"

In fact, I have created a second mapper with these settings, but I guess just changing the first one should be enough. If you want to check if the audience is set correctly, go to Clients -> netbird-client, go to tab "Client Scopes", subtab (one line below) "evaluate". Choose the user netbird and select "Generated access tokens" in the list on the right. Check if the "aud" claim contains "netbird-client".

[ergleb78]

We are experiencing the same issue on Google Auth. Management logs:

2024-06-27T16:56:41Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
2024-06-27T16:56:41Z ERRO management/server/http/middleware/auth_middleware.go:88: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-06-27T16:56:41Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
2024-06-27T16:56:41Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 798438459: GET /api/users status 401
2024-06-27T16:56:43Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
2024-06-27T16:56:43Z ERRO management/server/http/middleware/auth_middleware.go:88: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-06-27T16:56:43Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
2024-06-27T16:56:43Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 404750189: GET /api/users status 401

It's inconsistent: sometimes restart of docker-compose helps, sometimes it required to remove the containers and recreate

UPDATE: Downloading and updating GEO database fixed the issue.

Ask: It would be incredibly helpful to see some pointers to the root cause of the problem in the err logs.

[Vandaahl]

I had a problem with quite similar symptoms. This is possibly a bug in recent versions of keycloak, but I am not an expert with it.

However, I could work around the issue in the following way:

In the setup guide (https://docs.netbird.io/selfhosted/identity-providers#step-6-create-a-net-bird-client-scope), go to step 6 ("Create a NetBird client scope"). But instead of adding "netbird-client" to "Included Client Audience", add it to "Included Custom Audience"

In fact, I have created a second mapper with these settings, but I guess just changing the first one should be enough. If you want to check if the audience is set correctly, go to Clients -> netbird-client, go to tab "Client Scopes", subtab (one line below) "evaluate". Choose the user netbird and select "Generated access tokens" in the list on the right. Check if the "aud" claim contains "netbird-client".

Just wanted to say thank you for this comment. I wasted so many hours getting this to work with Keycloak and now it finally works :)

[Pshemas]

I have similar problem with Authentik. Initially the setup worked, but after restarting the containers for both Netbird and Authentik I get this dreaded 401: token invalid error.

After restarting the containers (both for Authentik and Netbird) I can't log in to management portal. Any suggestions what to do are greatly appreciated.

Error logs found in management component:

management-1  | 2024-07-02T15:39:41Z DEBG management/server/grpcserver.go:130: Sync request from peer [YvVW8g9sDDcUNhigOOW2SlIZBHj5Lj//mfMP2WAgzkg=] [56.67.17.123]
management-1  | 2024-07-02T15:39:41Z DEBG management/server/updatechannel.go:87: opened updates channel for a peer cpk3l2f7g7ts738pqbh0
management-1  | 2024-07-02T15:39:41Z DEBG management/server/telemetry/http_api_metrics.go:201: request OPTIONS /api/users took 0 ms and finished with status 204
management-1  | 2024-07-02T15:39:41Z DEBG management/server/account.go:1661: overriding JWT Domain and DomainCategory claims since single account mode is enabled
management-1  | 2024-07-02T15:39:41Z DEBG management/server/account.go:1810: Acquired global lock in 77.978µs for user 7
management-1  | 2024-07-02T15:39:42Z DEBG management/server/sql_store.go:194: took 12 ms to persist an account to the store
management-1  | 2024-07-02T15:39:42Z DEBG management/server/account.go:1296: looking up user 7 of account cpk3ikv7g7ts73c049h0 in cache
management-1  | 2024-07-02T15:39:42Z DEBG management/server/account.go:1234: account cpk3ikv7g7ts73c049h0 not found in cache, reloading
management-1  | 2024-07-02T15:39:42Z DEBG management/server/idp/authentik.go:134: requesting new jwt token for authentik idp manager
management-1  | 2024-07-02T15:39:42Z ERRO management/server/http/middleware/auth_middleware.go:88: Error when validating JWT claims: unable to get authentik token, statusCode 400
management-1  | 2024-07-02T15:39:42Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
management-1  | 2024-07-02T15:39:42Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2496285363: GET /api/users status 401
management-1  | 2024-07-02T15:39:42Z DEBG management/server/telemetry/http_api_metrics.go:201: request GET /api/users took 101 ms and finished with status 401
management-1  | 2024-07-02T15:39:42Z DEBG management/server/account.go:1661: overriding JWT Domain and DomainCategory claims since single account mode is enabled

Docker Compose file:

services:
  #UI dashboard
  dashboard:
    image: netbirdio/dashboard:v2.4.0
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    env_file:
      - /home/uslugi/.ENV/.nbird
    volumes:
      - netbird-letsencrypt:/etc/letsencrypt

  # Signal
  signal:
    image: netbirdio/signal:0.28.3
    restart: unless-stopped
    volumes:
      - netbird-signal:/var/lib/netbird
    ports:
      - 10000:80
  #      # port and command for Let's Encrypt validation
  #      - 443:443
  #    command: ["--letsencrypt-domain", "net.mysite.com", "--log-file", "console"]

  # Management
  management:
    image: netbirdio/management:0.28.3
    restart: unless-stopped
    depends_on:
      - dashboard
    volumes:
      - netbird-mgmt:/var/lib/netbird
      - netbird-letsencrypt:/etc/letsencrypt:ro
      - /home/uslugi/management.json:/etc/netbird/management.json:z
    ports:
      - 33073:443 #API port
  #    # command for Let's Encrypt validation without dashboard container
  #    command: ["--letsencrypt-domain", "net.mysite.com", "--log-file", "console"]
    command: [
      "--port", "443",
      "--log-file", "console",
      "--log-level", "debug",
      "--disable-anonymous-metrics=true",
      "--single-account-mode-domain=net.mysite.com",
      "--dns-domain=netbird.selfhosted"
      ]

  # Coturn
  coturn:
    image: coturn/coturn:4.6.2
    restart: unless-stopped
    domainname: net.mysite.com
    volumes:
      - /home/uslugi/turnserver.conf:/etc/turnserver.conf:z
    #      - ./privkey.pem:/etc/coturn/private/privkey.pem:ro
    #      - ./cert.pem:/etc/coturn/certs/cert.pem:ro
    network_mode: host
    command:
      - -c /etc/turnserver.conf

volumes:
  netbird-mgmt:
  netbird-signal:
  netbird-letsencrypt:

Environment variables file:

# Endpoints
NETBIRD_MGMT_API_ENDPOINT=https://net.mysite.com:33073
NETBIRD_MGMT_GRPC_API_ENDPOINT=https://net.mysite.com:33073
# OIDC
AUTH_AUDIENCE=wbuBlzoRj/c5sn/xVXk0omZULBGChzyoCAhAR1NLgzs=
AUTH_CLIENT_ID=wbuBlzoRj/c5sn/xVXk0omZULBGChzyoCAhAR1NLgzs=
AUTH_CLIENT_SECRET=
AUTH_AUTHORITY=https://auth.mysite.com/application/o/netbird/
USE_AUTH0=false
AUTH_SUPPORTED_SCOPES=openid profile email offline_access api
AUTH_REDIRECT_URI=
AUTH_SILENT_REDIRECT_URI=
NETBIRD_TOKEN_SOURCE=accessToken
# SSL
NGINX_SSL_PORT=443
# Letsencrypt
LETSENCRYPT_DOMAIN=net.mysite.com
LETSENCRYPT_EMAIL=myemail@mysite.com

I've also checked Authentik and I found "application authorized" event:

{
    "user": {
        "pk": 7,
        "email": "mymail@mydomain.com",
        "username": "myusername"
    },
    "action": "authorize_application",
    "app": "authentik.providers.oauth2.views.authorize",
    "context": {
        "flow": "82fcc99a48664ec494ce06c38327c3b7",
        "scopes": "profile email openid",
        "http_request": {
            "args": {
                "scope": "openid profile email offline_access api",
                "state": "hdKCFMF3p9",
                "audience": "someid",
                "client_id": "someid",
                "redirect_uri": "https://net.mydomain.com/#callback",
                "response_type": "code",
                "code_challenge": "somecodechallenge",
                "code_challenge_method": "S256"
            },
            "path": "/api/v3/flows/executor/default-provider-authorization-explicit-consent/",
            "method": "GET",
            "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
        },
        "authorized_application": {
            "pk": "somepk",
            "app": "authentik_core",
            "name": "Netbird",
            "model_name": "application"
        }
    },
    "client_ip": "some.ip",
    "expires": "2025-07-02T16:12:31.981Z",
    "brand": {
        "pk": "somepk",
        "app": "authentik_brands",
        "name": "Default brand",
        "model_name": "brand"
    }
}

[vsavovski]

Regarding Keycloak, it is possible to use the original setup; however, you cannot provide netbird-client as the ID. Instead, you must use the generated GUID that Keycloak creates for each client ID.

You can find the GUID in the URL: https://keycloak.mysite.com/admin/master/console/#/{realm}/clients/{client-id}.

Alternatively, as @HansAschauer mentioned, you can generate access tokens and locate the client ID in the aud field. This client ID should then be used in the .env file.

[Pshemas]

so far I've tried:

• NETBIRD_MGMT_IDP_SIGNKEY_REFRESH=true added to .env

https://github.com/netbirdio/netbird/issues/1531#issuecomment-1929102315

• copied geo database

https://github.com/netbirdio/netbird/issues/2142#issuecomment-2172029587

This doesn't help sadly. IWhen I look into developer console I see this:

And token invaild in netbird management logs as posted above.

I wonder - can it be something with Authentik being behind Cloudflare? But on the other hand it does not cause any issues on other apps I use with Authentik (and the super annoying thing is that it worked for a couple of weeks without a hitch).

[Pshemas]

on my end it "autmagically" started working - thus suggesting something to do with Authentik config, not Netbird itself.

[identw]

I have the same problem but with Dex. Right after starting netbird-management, everything works and the login succeeds. However, if I wait a couple of hours, I get the same error upon login:

2024-07-04T20:36:48Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
2024-07-04T20:36:48Z ERRO management/server/http/middleware/auth_middleware.go:88: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-07-04T20:36:48Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
2024-07-04T20:36:48Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 4269949527: GET /api/users status 401

[mlsmaycon]

I have the same problem but with Dex. Right after starting netbird-management, everything works and the login succeeds. However, if I wait a couple of hours, I get the same error upon login:

2024-07-04T20:36:48Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
2024-07-04T20:36:48Z ERRO management/server/http/middleware/auth_middleware.go:88: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-07-04T20:36:48Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
2024-07-04T20:36:48Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 4269949527: GET /api/users status 401

@identw you need to enable sign key refresh with --idp-sign-key-refresh-enabled

    command: [
     ...
      "--idp-sign-key-refresh-enabled",

After that run docker compose up -d

[identw]

@mlsmaycon Thank you very much. This helped me

[singhera-ilmiya]

I'm still getting same issue please help me @mlsmaycon @identw

[mlsmaycon]

@singhera-ilmiya can you check your management logs for error logs and share them with us?

[bl0way]

Regarding Keycloak, it is possible to use the original setup; however, you cannot provide netbird-client as the ID. Instead, you must use the generated GUID that Keycloak creates for each client ID.

You can find the GUID in the URL: https://keycloak.mysite.com/admin/master/console/#/{realm}/clients/{client-id}.

Alternatively, as @HansAschauer mentioned, you can generate access tokens and locate the client ID in the aud field. This client ID should then be used in the .env file.

Indeed, this worked for me. The previous proposed solution unfortunately was not working for me (keycloak didn't add the provided Included Custom Audience in the generated token for X or Y reasons). I modified the management.json to update the AuthAudience by the generated GUID of keycloak.

[adriangabura]

Is it possible these 401 issues are related to this? I have triple checked my Azure config. There is no error on my part. And far too many identity providers cause similar symptoms.

[loso2255]

i think i found the same problem with zitadel

note: i'm using the latest version, docker stack

2024-07-20T18:04:13Z ERRO [requestID: 0c185520-9b12-49bc-bbb8-3589329c90d4, context: HTTP] management/server/http/util/util.go:81: got a handler error: token invalid
2024-07-20T18:04:13Z ERRO [context: HTTP, requestID: 0c185520-9b12-49bc-bbb8-3589329c90d4] management/server/telemetry/http_api_metrics.go:191: HTTP response 0c185520-9b12-49bc-bbb8-3589329c90d4: GET /api/users status 401
2024-07-20T18:04:30Z ERRO [context: HTTP, requestID: 70ea143e-a6f2-4963-ab18-e83d31a247e1] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get zitadel token, statusCode 400

[mannp]

For me it was resolved by changing the type of jwt signature being used.

[maslyankov]

For me this issue was fixed with this: https://github.com/netbirdio/netbird/issues/2142#issuecomment-2182390323

Basically In keycloak I configured :

Client scopes > Client scope details > Mapper details Move "netbird-client" from "Included Client Audience ()" to "Included Custom Audience" leaving "Included Client Audience ()" .