Can't access dashboard - Token Invalid, Authentik

[Pshemas]

I've been looking at similar reports - and I couldn't figure out which one would be best for this one, in the end decided on new one, hopefully all the appropriate ones will be merged.

So I had a working self hosted instance of Netbird with Authentik as a IdP provider. After a while it stopped working with Token Invalid error message... which "magically" fixed itself. But now it stopped working again and I can't access the dashboard (the service itself works, the agents can connect, but I can't do any management atm).

Here's what I see in the logs:

2024-07-28T12:35:58Z DEBG management/server/idp/authentik.go:134: requesting new jwt token for authentik idp manager
2024-07-28T12:35:58Z ERRO [context: HTTP, requestID: a7655341-9864-4855-b005-3fa72ca9b82a] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400
2024-07-28T12:35:58Z ERRO [context: HTTP, requestID: a7655341-9864-4855-b005-3fa72ca9b82a] management/server/http/util/util.go:81: got a handler error: token invalid
2024-07-28T12:35:58Z ERRO [requestID: a7655341-9864-4855-b005-3fa72ca9b82a, context: HTTP] management/server/telemetry/http_api_metrics.go:191: HTTP response a7655341-9864-4855-b005-3fa72ca9b82a: GET /api/users status 401
2024-07-28T12:35:58Z DEBG [context: HTTP, requestID: a7655341-9864-4855-b005-3fa72ca9b82a] management/server/telemetry/http_api_metrics.go:211: request GET /api/users took 305 ms and finished with status 401
2024-07-28T12:35:59Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:112: keys refreshed, new UTC expiration time: 2024-07-28 12:35:59.293866388 +0000 UTC
2024-07-28T12:35:59Z DEBG [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/account.go:1667: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2024-07-28T12:35:59Z DEBG [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/account.go:1816: Acquired global lock in 8.327µs for user 7
2024-07-28T12:35:59Z DEBG [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/sql_store.go:169: took 8 ms to persist an account to the store
2024-07-28T12:35:59Z DEBG [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/account.go:1301: looking up user 7 of account cpk3ikv7g7ts73c049h0 in cache
2024-07-28T12:35:59Z DEBG management/server/account.go:1239: account cpk3ikv7g7ts73c049h0 not found in cache, reloading
2024-07-28T12:35:59Z DEBG management/server/idp/authentik.go:134: requesting new jwt token for authentik idp manager
2024-07-28T12:35:59Z ERRO [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400
2024-07-28T12:35:59Z ERRO [requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418, context: HTTP] management/server/http/util/util.go:81: got a handler error: token invalid
2024-07-28T12:35:59Z ERRO [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/telemetry/http_api_metrics.go:191: HTTP response 859af32c-cfd2-4633-a3b1-2c2bba6b0418: GET /api/users status 401
2024-07-28T12:35:59Z DEBG [context: HTTP, requestID: 859af32c-cfd2-4633-a3b1-2c2bba6b0418] management/server/telemetry/http_api_metrics.go:211: request GET /api/users took 321 ms and finished with status 401

Here's sanitized management.json:

{
    "Stuns": [{
        "Proto": "udp",
        "URI": "stun:mydomain.com:3478",
        "Username": "",
        "Password": ""
    }],
    "TURNConfig": {
        "TimeBasedCredentials": false,
        "CredentialsTTL": "12h0m0s",
        "Secret": "secret",
        "Turns": [{
            "Proto": "udp",
            "URI": "turn:mydomain.com:3478",
            "Username": "self",
            "Password": "someturnpassword"
        }]
    },
    "Signal": {
        "Proto": "http",
        "URI": "mydomain.com:10000",
        "Username": "",
        "Password": ""
    },
    "Datadir": "/var/lib/netbird/",
    "DataStoreEncryptionKey": "somekey",
    "HttpConfig": {
        "LetsEncryptDomain": "",
        "CertFile": "/etc/letsencrypt/live/mydomain.com/fullchain.pem",
        "CertKey": "/etc/letsencrypt/live/mydomain.com/privkey.pem",
        "AuthAudience": "OauthProiderClientID",
        "AuthIssuer": "https://authentik.mydomain.com/application/o/netbird/",
        "AuthUserIDClaim": "",
        "AuthKeysLocation": "https://authentik.mydomain.com/application/o/netbird/jwks/",
        "OIDCConfigEndpoint": "https://authentik.mydomain.com/application/o/netbird/.well-known/openid-configuration",
        "IdpSignKeyRefreshEnabled": true
    },
    "IdpManagerConfig": {
        "ManagerType": "authentik",
        "ClientConfig": {
            "Issuer": "https://authentik.mydomain.com/application/o/netbird",
            "TokenEndpoint": "https://authentik.mydomain.com/application/o/token/",
            "ClientID": "OauthProiderClientID",
            "ClientSecret": "",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "Password": "ServiceAccountToken",
            "Username": "Netbird"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },
    "DeviceAuthorizationFlow": {
        "Provider": "hosted",
        "ProviderConfig": {
            "ClientID": "OauthProiderClientID",
            "ClientSecret": "",
            "Domain": "authentik.mydomain.com",
            "Audience": "OauthProiderClientID",
            "TokenEndpoint": "https://authentik.mydomain.com/application/o/token/",
            "DeviceAuthEndpoint": "https://authentik.mydomain.com/application/o/device/",
            "AuthorizationEndpoint": "",
            "Scope": "openid",
            "UseIDToken": false,
            "RedirectURLs": null
        }
    },
    "PKCEAuthorizationFlow": {
        "ProviderConfig": {
            "ClientID": "OauthProiderClientID",
            "ClientSecret": "",
            "Domain": "",
            "Audience": "OauthProiderClientID",
            "TokenEndpoint": "https://authentik.mydomain.com/application/o/token/",
            "DeviceAuthEndpoint": "",
            "AuthorizationEndpoint": "https://authentik.mydomain.com/application/o/authorize/",
            "Scope": "openid profile email offline_access api",
            "UseIDToken": false,
            "RedirectURLs": [
                "http://localhost:53000"
            ]
        }
    },
    "StoreConfig": {
        "Engine": "sqlite"
    },
    "ReverseProxy": {
        "TrustedHTTPProxies": [],
        "TrustedHTTPProxiesCount": 0,
        "TrustedPeers": [
            "0.0.0.0/0"
        ]
    }
}

Here's sanitized openid config:

{
    "issuer": "https://authentik.mydomain.com/application/o/netbird/",
    "authorization_endpoint": "https://authentik.mydomain.com/application/o/authorize/",
    "token_endpoint": "https://authentik.mydomain.com/application/o/token/",
    "userinfo_endpoint": "https://authentik.mydomain.com/application/o/userinfo/",
    "end_session_endpoint": "https://authentik.mydomain.com/application/o/netbird/end-session/",
    "introspection_endpoint": "https://authentik.mydomain.com/application/o/introspect/",
    "revocation_endpoint": "https://authentik.mydomain.com/application/o/revoke/",
    "device_authorization_endpoint": "https://authentik.mydomain.com/application/o/device/",
    "response_types_supported": [
        "code",
        "id_token",
        "id_token token",
        "code token",
        "code id_token",
        "code id_token token"
    ],
    "response_modes_supported": [
        "query",
        "fragment",
        "form_post"
    ],
    "jwks_uri": "https://authentik.mydomain.com/application/o/netbird/jwks/",
    "grant_types_supported": [
        "authorization_code",
        "refresh_token",
        "implicit",
        "client_credentials",
        "password",
        "urn:ietf:params:oauth:grant-type:device_code"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "subject_types_supported": [
        "public"
    ],
    "token_endpoint_auth_methods_supported": [
        "client_secret_post",
        "client_secret_basic"
    ],
    "acr_values_supported": [
        "goauthentik.io/providers/oauth2/default"
    ],
    "scopes_supported": [
        "email",
        "profile",
        "openid"
    ],
    "request_parameter_supported": false,
    "claims_supported": [
        "sub",
        "iss",
        "aud",
        "exp",
        "iat",
        "auth_time",
        "acr",
        "amr",
        "nonce",
        "email",
        "email_verified",
        "name",
        "given_name",
        "preferred_username",
        "nickname",
        "groups"
    ],
    "claims_parameter_supported": false,
    "code_challenge_methods_supported": [
        "plain",
        "S256"
    ]
}

Netbird is running inside Docker container, while Authentik in Podman one, on a separate server (with Caddy reverse proxy and Cloudflare).

I'm using Authentik for several other apps and I don't have any issues there (but there's one difference - for other apps I don't use service account setup).

On the side of Authentik I don't see any problems. Here's raw event info:

{
    "user": {
        "pk": 7,
        "email": "myemail",
        "username": "myusername"
    },
    "action": "authorize_application",
    "app": "authentik.providers.oauth2.views.authorize",
    "context": {
        "flow": "someflow",
        "scopes": "offline_access openid email profile",
        "http_request": {
            "args": {
                "scope": "openid profile email offline_access api",
                "state": "7Cwo6bqD1f",
                "audience": "OauthProviderClientID",
                "client_id": "OauthProviderClientID",
                "redirect_uri": "https://mydomain.com/#callback",
                "response_type": "code",
                "code_challenge": "somechallenge",
                "code_challenge_method": "S256"
            },
            "path": "/api/v3/flows/executor/default-provider-authorization-explicit-consent/",
            "method": "GET",
            "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
        },
        "authorized_application": {
            "pk": "somepk",
            "app": "authentik_core",
            "name": "Netbird",
            "model_name": "application"
        }
    },
    "client_ip": "someip",
    "expires": "2025-07-28T12:51:42.272Z",
    "brand": {
        "pk": "somepk",
        "app": "authentik_brands",
        "name": "Default brand",
        "model_name": "brand"
    }
}

In credentials / tokens for a user that wishes to access Netbird I see:

Here are provider settings:

Any suggestions howto resolve the issue and get into the management panel are greatly appreciated. At this point I'm just blindly clicking various options as the suggestions in other topics are all over the place - it seems that I'm not the only one who has issues in pinpointing the cause / fix.

If there's some more info needed plz let me know - I'll be happy to provide it.

[Pshemas]

for the time being I've created a new provider and service account to get into the dashboard, but I fully expect the problem to reappear when token expires.