Netbird doesn't work with latest Zitadel (v2.42.2) Invalid JWT

[smartyfeed]

Describe the problem Management component can't communicate with latest Zitadel version. I followed the Advanced install guide and tried to setup Netbird with an already existing instance of Zitadel (latest version). After logging in I get Invalid JWT error and error messages in logs: 2023-12-17T22:31:18Z ERRO management/server/http/middleware/access_control.go:46: failed to get user from claims: failed to get account with token claims unable to post https://_domain redacted_:443/management/v1/users/_search, statusCode 401

To Reproduce Steps to reproduce the behavior:

1. Download getting-started-with-zitadel.sh script.
2. Modify the version of Zitadel and CockroachDB in the script to latest.
3. Execute the script
4. Check management container logs and see error

Expected behavior Normal startup of Netbird

Additional context Zitadel reports successful secret check. If you leave Zitadel version as is, Netbird boots up fine

[mlsmaycon]

Hello @smartyfeed, thanks for reporting the issue. We will check if there was a breaking change in the API endpoints used by the IDP manager.

[smartyfeed]

@bcmmbaga @mlsmaycon, I tested the proposed solution and can confirm it makes it work with the latest Zitadel. Thanks for the speedy fix!

However, issue still persists when trying to install using the advanced setup guide. Error has changed to

Error logs: artifacts-management-1 | 2023-12-19T00:53:54Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_domain redacted_/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T00:53:54Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid

[bcmmbaga]

Hi @smartyfeed , Could you please share the value of TokenEndpoint in management.json under IdpManagerConfig > ClientConfig? Be sure to redact any sensitive information, but please do not remove the port. Did you generate the configuration artifacts using setup.env with the configure.sh script?

[smartyfeed]

Hi @bcmmbaga, i modified the configure script in the same manner as you have modified the getting started script, removed all mentions of port from the endpoints. Therefor currently token endpoint looks like this: https://_domain without a port_/oauth/v2/token

[bcmmbaga]

Hi @bcmmbaga, i modified the configure script in the same manner as you have modified the getting started script, removed all mentions of port from the endpoints. Therefor currently token endpoint looks like this: https://_domain without a port_/oauth/v2/token

I don't think you need to modify the configure script. You can follow these steps to set up with Zitadel manually. But if that's not the case, it's better to use the getting started with Zitadel script to set it up.

[smartyfeed]

Hi @bcmmbaga, i modified the configure script in the same manner as you have modified the getting started script, removed all mentions of port from the endpoints. Therefor currently token endpoint looks like this: https://_domain without a port_/oauth/v2/token

I don't think you need to modify the configure script. You can follow these steps to set up with Zitadel manually. But if that's not the case, it's better to use the getting started with Zitadel script to set it up.

I will try to set it up using exactly that guide. However, will it work with externally hosted Zitadel? I encountered the Invalid JWT error when tying to connect to my existing Zitadel instance.

[bcmmbaga]

Hi @bcmmbaga, i modified the configure script in the same manner as you have modified the getting started script, removed all mentions of port from the endpoints. Therefor currently token endpoint looks like this: https://_domain without a port_/oauth/v2/token

I don't think you need to modify the configure script. You can follow these steps to set up with Zitadel manually. But if that's not the case, it's better to use the getting started with Zitadel script to set it up.

I will try to set it up using exactly that guide. However, will it work with externally hosted Zitadel? I encountered the Invalid JWT error when tying to connect to my existing Zitadel instance.

Yes, it should work with an externally hosted Zitadel if you follow the steps on how to set it up correctly, as specified in the above link.

[smartyfeed]

Hi @bcmmbaga, just tried to install following Advanced Guide exactly. Unfortunately issue still persists.

Error logs: artifacts-dashboard-1 | 172.22.0.1 - - [19/Dec/2023:12:29:43 +0000] "GET /static/js/main.b25ec689.js HTTP/1.0" 304 0 "https://_domain redacted_/auth?code=agnF0Fp7Qi2oMz5geaqMH6AKFeD_9oZLRTqKernow5P-vQ&state=4wo6eHbUWY" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0" "ip redacted" artifacts-management-1 | 2023-12-19T12:29:43Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_domain redacted_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 190259666: GET /api/users status 401 artifacts-dashboard-1 | 172.22.0.1 - - [19/Dec/2023:12:29:43 +0000] "GET /static/media/logo.36ccce29c76d193b4175.png HTTP/1.0" 304 0 "https://_domain redacted_/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0" "ip redacted" artifacts-dashboard-1 | 172.22.0.1 - - [19/Dec/2023:12:29:43 +0000] "GET /static/media/logo.36ccce29c76d193b4175.png HTTP/1.0" 304 0 "https://_domain redacted_/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0" "ip redacted" artifacts-management-1 | 2023-12-19T12:29:43Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T12:29:43Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T12:29:43Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_domain redacted_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3906661011: GET /api/users status 401 artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_domain redacted_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 491526805: GET /api/users status 401 artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_domain redacted_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T12:29:43Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2389077942: GET /api/groups status 401

[bcmmbaga]

@smartyfeed, can you share the management.json file, but please remember to redact any sensitive information, leaving the port number visible?

[smartyfeed]

@smartyfeed, can you share the management.json file, but please remember to redact any sensitive information, leaving the port number visible?

Sure, redacted all the data but left general meaning. vpn.example.com is the domain Netbird is running at and zitadel.example.com is the domain Zitadel is hosted under.

{
    "Stuns": [
        {
            "Proto": "udp",
            "URI": "stun:_vpn.example.com_:3478",
            "Username": "",
            "Password": ""
        }
    ],
    "TURNConfig": {
        "TimeBasedCredentials": false,
        "CredentialsTTL": "12h0m0s",
        "Secret": "secret",
        "Turns": [
            {
                "Proto": "udp",
                "URI": "turn:_vpn.example.com_:3478",
                "Username": "self",
                "Password": "_TURN Password_"
            }
        ]
    },
    "Signal": {
        "Proto": "https",
        "URI": "_vpn.example.com_:443",
        "Username": "",
        "Password": ""
    },
    "Datadir": "/var/lib/netbird/",
    "DataStoreEncryptionKey": "_DataStoreEncryptionKey_",
    "HttpConfig": {
        "LetsEncryptDomain": "",
        "CertFile": "",
        "CertKey": "",
        "AuthAudience": "_Client ID_",
        "AuthIssuer": "https://_zitadel.example.com_:443",
        "AuthUserIDClaim": "",
        "AuthKeysLocation": "https://_zitadel.example.com_:443/oauth/v2/keys",
        "OIDCConfigEndpoint": "https://_zitadel.example.com_:443/.well-known/openid-configuration",
        "IdpSignKeyRefreshEnabled": false
    },
    "IdpManagerConfig": {
        "ManagerType": "zitadel",
        "ClientConfig": {
            "Issuer": "https://_zitadel.example.com_:443",
            "TokenEndpoint": "https://_zitadel.example.com_:443/oauth/v2/token",
            "ClientID": "netbird",
            "ClientSecret": "_Client secret_",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "ManagementEndpoint": "https://_zitadel.example.com_:443/management/v1"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },
    "DeviceAuthorizationFlow": {
        "Provider": "hosted",
        "ProviderConfig": {
            "ClientID": "_Client ID_",
            "ClientSecret": "",
            "Domain": "_zitadel.example.com_:443",
            "Audience": "_Client ID_",
            "TokenEndpoint": "https://_zitadel.example.com_:443/oauth/v2/token",
            "DeviceAuthEndpoint": "https://_zitadel.example.com_:443/oauth/v2/device_authorization",
            "AuthorizationEndpoint": "",
            "Scope": "openid",
            "UseIDToken": false,
            "RedirectURLs": null
        }
    },
    "PKCEAuthorizationFlow": {
        "ProviderConfig": {
            "ClientID": "_Client ID_",
            "ClientSecret": "",
            "Domain": "",
            "Audience": "_Client ID_",
            "TokenEndpoint": "https://_zitadel.example.com_:443/oauth/v2/token",
            "DeviceAuthEndpoint": "",
            "AuthorizationEndpoint": "https://_zitadel.example.com_:443/oauth/v2/authorize",
            "Scope": "openid profile email offline_access api",
            "UseIDToken": false,
            "RedirectURLs": [
                "http://localhost:53000"
            ]
        }
    },
    "StoreConfig": {
        "Engine": "jsonfile"
    }
}

[bcmmbaga]

"IdpManagerConfig": {
        "ManagerType": "zitadel",
        "ClientConfig": {
            "Issuer": "https://_zitadel.example.com_:443",
            "TokenEndpoint": "https://_zitadel.example.com_:443/oauth/v2/token",
            "ClientID": "netbird",
            "ClientSecret": "_Client secret_",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "ManagementEndpoint": "https://_zitadel.example.com_:443/management/v1"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },

Was this generated by configure.sh or getting-started-with-zitadel.sh? Please edit to remove the port 443 in IdpManagerConfig > ClientConfig > issuer and IdpManagerConfig > ClientConfig > TokenEndpoint, and then restart the management service with the command docker-compose down management && docker-compose up -d management.

[smartyfeed]

"IdpManagerConfig": {
        "ManagerType": "zitadel",
        "ClientConfig": {
            "Issuer": "https://_zitadel.example.com_:443",
            "TokenEndpoint": "https://_zitadel.example.com_:443/oauth/v2/token",
            "ClientID": "netbird",
            "ClientSecret": "_Client secret_",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "ManagementEndpoint": "https://_zitadel.example.com_:443/management/v1"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },

Was this generated by configure.sh or getting-started-with-zitadel.sh? Please edit to remove the port 443 in IdpManagerConfig > ClientConfig > issuer and IdpManagerConfig > ClientConfig > TokenEndpoint, and then restart the management service with the command docker-compose down management && docker-compose up -d management.

That config was generated with configure.sh. After removing the port and restarting the management container the issue is still present

[bcmmbaga]

@smartyfeed, update the management service in docker-compose.yml to include --log-level debug in the commands, then run the commands docker-compose down --volumes and docker-compose up -d, and finally, share the management logs again.

[smartyfeed]

@smartyfeed, update the management service in docker-compose.yml to include --log-level debug in the commands, then run the commands docker-compose down --volumes and docker-compose up -d, and finally, share the management logs again.

Here you go: artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:422: loading OIDC configuration from the provided IDP configuration endpoint https://_zitadel.example.com_:443/.well-known/openid-configuration artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:427: loaded OIDC configuration from the provided IDP configuration endpoint: https://_zitadel.example.com_:443/.well-known/openid-configuration artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:429: overriding HttpConfig.AuthIssuer with a new value https://_zitadel.example.com_:443, previously configured value: https://_zitadel.example.com_:443 artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:433: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://_zitadel.example.com_:443/oauth/v2/keys, previously configured value: https://_zitadel.example.com_:443/oauth/v2/keys artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:438: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/token, previously configured value: https://_zitadel.example.com_:443/oauth/v2/token artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:441: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/device_authorization, previously configured value: https://_zitadel.example.com_:443/oauth/v2/device_authorization artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:449: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: zitadel.example.com:443, previously configured value: zitadel.example.com:443 artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:459: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/token, previously configured value: https://_zitadel.example.com_:443/oauth/v2/token artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:462: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/authorize, previously configured value: https://_zitadel.example.com_:443/oauth/v2/authorize artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081 artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/server/store.go:74: using JSON file store engine artifacts-management-1 | 2023-12-19T13:02:18Z DEBG management/server/file_store.go:269: took 0 ms to persist the FileStore artifacts-management-1 | 2023-12-19T13:02:18Z DEBG management/server/activity/sqlite/sqlite.go:328: check deleted_users table version artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/server/account.go:828: single account mode enabled, accounts number 0 artifacts-management-1 | 2023-12-19T13:02:18Z DEBG management/server/idp/zitadel.go:145: requesting new jwt token for zitadel idp manager artifacts-management-1 | 2023-12-19T13:02:18Z DEBG management/server/ephemeral.go:135: loaded ephemeral peer(s): 0 artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:250: running gRPC backward compatibility server: [::]:33073 artifacts-management-1 | 2023-12-19T13:02:18Z INFO management/cmd/management.go:282: running HTTP server and gRPC server on the same port: [::]:443 artifacts-management-1 | 2023-12-19T13:02:19Z WARN management/server/account.go:868: failed warming up cache due to error: unable to post https://_zitadel.example.com_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/jwtclaims/jwtValidator.go:220: generating validation pem from JWK artifacts-management-1 | 2023-12-19T13:02:59Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:275: acquiring global lock artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:285: took 640ns to acquire global lock artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1770: creating new account artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1782: created new account cm0p90v46orc73aoramg artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:269: took 0 ms to persist the FileStore artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1210: looking up user 245130045285386445 of account cm0p90v46orc73aoramg in cache artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1148: account cm0p90v46orc73aoramg not found in cache, reloading artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:281: released global lock in 30.680666ms artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_zitadel.example.com_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3426305082: GET /api/users status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/telemetry/http_api_metrics.go:201: request GET /api/users took 31 ms and finished with status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/jwtclaims/jwtValidator.go:220: generating validation pem from JWK artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/jwtclaims/jwtValidator.go:220: generating validation pem from JWK artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/jwtclaims/jwtValidator.go:220: generating validation pem from JWK artifacts-management-1 | 2023-12-19T13:02:59Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:275: acquiring global lock artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:285: took 640ns to acquire global lock artifacts-management-1 | 2023-12-19T13:02:59Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:275: acquiring global lock artifacts-management-1 | 2023-12-19T13:02:59Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:275: acquiring global lock artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:269: took 0 ms to persist the FileStore artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1210: looking up user 245130045285386445 of account cm0p90v46orc73aoramg in cache artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1148: account cm0p90v46orc73aoramg not found in cache, reloading artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:281: released global lock in 27.598155ms artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_zitadel.example.com_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 745840097: GET /api/users status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:285: took 27.478836ms to acquire global lock artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/telemetry/http_api_metrics.go:201: request GET /api/users took 28 ms and finished with status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:269: took 0 ms to persist the FileStore artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1210: looking up user 245130045285386445 of account cm0p90v46orc73aoramg in cache artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1148: account cm0p90v46orc73aoramg not found in cache, reloading artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:281: released global lock in 55.794909ms artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_zitadel.example.com_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2655761295: GET /api/groups status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:285: took 55.811909ms to acquire global lock artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/telemetry/http_api_metrics.go:201: request GET /api/groups took 56 ms and finished with status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:269: took 0 ms to persist the FileStore artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1210: looking up user 245130045285386445 of account cm0p90v46orc73aoramg in cache artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/account.go:1148: account cm0p90v46orc73aoramg not found in cache, reloading artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/file_store.go:281: released global lock in 89.327046ms artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to post https://_zitadel.example.com_:443/management/v1/users/_search, statusCode 401 artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid artifacts-management-1 | 2023-12-19T13:02:59Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3063729297: GET /api/users status 401 artifacts-management-1 | 2023-12-19T13:02:59Z DEBG management/server/telemetry/http_api_metrics.go:201: request GET /api/users took 90 ms and finished with status 401

[bcmmbaga]

@smartyfeed Can you remove the 443 port in NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT in your setup.env, generate the configuration with configure.sh, and then run the commands docker-compose down and docker-compose up -d, and finally, test again?

[smartyfeed]

docker-compose up -d

Removed the port from NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT, regenerated the config and replaced ports in docker-compose.yml

Here are the logs: artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:422: loading OIDC configuration from the provided IDP configuration endpoint https://_zitadel.example.com_/.well-known/openid-configuration artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:427: loaded OIDC configuration from the provided IDP configuration endpoint: https://_zitadel.example.com_/.well-known/openid-configuration artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:429: overriding HttpConfig.AuthIssuer with a new value https://_zitadel.example.com_:443, previously configured value: https://_zitadel.example.com_:443 artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:433: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://_zitadel.example.com_:443/oauth/v2/keys, previously configured value: https://_zitadel.example.com_:443/oauth/v2/keys artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:438: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/token, previously configured value: https://_zitadel.example.com_:443/oauth/v2/token artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:441: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/device_authorization, previously configured value: https://_zitadel.example.com_:443/oauth/v2/device_authorization artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:449: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: zitadel.example.com, previously configured value: artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:459: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/token, previously configured value: https://_zitadel.example.com_:443/oauth/v2/token artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:462: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://_zitadel.example.com_:443/oauth/v2/authorize, previously configured value: https://_zitadel.example.com_:443/oauth/v2/authorize artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081 artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/server/store.go:74: using JSON file store engine artifacts-management-1 | 2023-12-19T13:43:11Z DEBG management/server/file_store.go:269: took 0 ms to persist the FileStore artifacts-management-1 | 2023-12-19T13:43:11Z DEBG management/cmd/management.go:307: generate new activity store encryption key artifacts-management-1 | 2023-12-19T13:43:11Z DEBG management/server/activity/sqlite/sqlite.go:328: check deleted_users table version artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:155: update config with activity store key artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/server/account.go:828: single account mode enabled, accounts number 1 artifacts-management-1 | 2023-12-19T13:43:11Z DEBG management/server/idp/zitadel.go:145: requesting new jwt token for zitadel idp manager artifacts-management-1 | 2023-12-19T13:43:11Z DEBG management/server/ephemeral.go:135: loaded ephemeral peer(s): 0 artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:250: running gRPC backward compatibility server: [::]:33073 artifacts-management-1 | 2023-12-19T13:43:11Z INFO management/cmd/management.go:282: running HTTP server and gRPC server on the same port: [::]:443 artifacts-management-1 | 2023-12-19T13:43:12Z WARN management/server/account.go:868: failed warming up cache due to error: unable to post https://_zitadel.example.com_:443/management/v1/users/_search, statusCode 401

[bcmmbaga]

replaced ports in docker-compose.yml

Do not replace ports in docker-compose.yml. Re-generate the configurations and then run the Docker commands without changing anything else.

[smartyfeed]

replaced ports in docker-compose.yml

Do not replace ports in docker-compose.yml. Re-generate the configurations and then run the Docker commands without changing anything else.

I am using a reverse proxy, so some services can't start as the port 443 is already in use by Nginx

[bcmmbaga]

replaced ports in docker-compose.yml

Do not replace ports in docker-compose.yml. Re-generate the configurations and then run the Docker commands without changing anything else.

I am using a reverse proxy, so some services can't start as the port 443 is already in use by Nginx

Okay, for that case, you can change it, but I need you to confirm something with me. Can you go to the OIDC endpoint, take note of the issuer value, and compare it with the one in management.json? Check if IdpManagerConfig > ClientConfig > issuer matches

[smartyfeed]

replaced ports in docker-compose.yml

Do not replace ports in docker-compose.yml. Re-generate the configurations and then run the Docker commands without changing anything else.

I am using a reverse proxy, so some services can't start as the port 443 is already in use by Nginx

Okay, for that case, you can change it, but I need you to confirm something with me. Can you go to the OIDC endpoint, take note of the issuer value, and compare it with the one in management.json? Check if IdpManagerConfig > ClientConfig > issuer matches

Hey, @bcmmbaga. Just checked the issuer and it matches. Yesterday I tried to debug this problem a bit more and tried to connect Netbird to different versions of Zitadel on a fresh VPS. This issue exists for several versions, latest version I successfully got connected to was v2.38.1

[bcmmbaga]

@smartyfeed, I will look into this in detail. In the meantime, could you consider joining our Slack channel? I'll be more helpful there for further debugging.

[smartyfeed]

@smartyfeed, I will look into this in detail. In the meantime, could you consider joining our Slack channel? I'll be more helpful there for further debugging.

Thank you! Joined your Slack:)

[TSJasonH]

Ran across this and wondered if there is any relationship with https://github.com/goauthentik/authentik/issues/7907 which was opened only a day after this one. Seems the conversation was moved to slack so unsure of the progress or resolution.

[landmass-deftly-reptile-budget]

Hi, can you maybe give us a hint here in the comments how you did solve this issue or what the root cause was? Many thanks.

[smartyfeed]

Hi, I gave up Netbird for now and did not perform any more testing on my side. My only testing steps are mentioned in this issue. Unfortunately, I don't know if the issue has been fixed

[mlsmaycon]

Hello folks, the issue has been fixed since v0.25.3, if you still facing issues, please share with us your Zitadel and management logs

[landmass-deftly-reptile-budget]

Thanks for the hint that it was a "bug" and had been fixed since v0.25.3. In consequence it had to be some wrong configuration.... which it was and I finally found it. For other people who run in this issue, at least what worked for me is: "IdpSignKeyRefreshEnabled": true This value has to be true in management.json

[mighty-services]

IdpSignKeyRefreshEnabled

Dear landmass-deftly-reptile-budget, where did you change/put this in?

I´m having the same issues but can´t find it.

I´ve started with the quistart script (zitadel "integrated so to say) and moved to cloning this repo and put all variables like in the first setup. But stil i can´t get it working - neither way

[landmass-deftly-reptile-budget]

IdpSignKeyRefreshEnabled

Dear landmass-deftly-reptile-budget, where did you change/put this in?

I´m having the same issues but can´t find it.

I´ve started with the quistart script (zitadel "integrated so to say) and moved to cloning this repo and put all variables like in the first setup. But stil i can´t get it working - neither way

In the management.json file at the topic:

"HttpConfig": {   "LetsEncryptDomain": "",   "CertFile": "",   "CertKey": "",   "AuthAudience": "XXX",   "AuthIssuer": "https://whatever.com",   "AuthUserIDClaim": "",   "AuthKeysLocation": "https://lwhatever.com/oauth/v2/keys",   "OIDCConfigEndpoint": "https://lwhatever.com/.well-known/openid-configuration",   "IdpSignKeyRefreshEnabled": true   },

[mighty-services]

Thanks landmass-deftly-reptile-budget for your quick reply.

I looked in my management.json file, and it was already present and set to true. I guess this was implemented with the fix in v0.25.3 mlsmaycon mentioned.

I've reset it to the last working backup right after the initial installation. Furthermore, I'll test and see how it behaves with me doing something in zitadel - I assume it's a layer 8 problem and I messed with zitadel, so that it's not working anymore.

[disambiguation]

After more time than I would like to admit, I resolved my problem that presented itself just like in this issue. Figured this might help someone else if they have the same configuration. My problem wasn't with Netbird configuration, it was with the reverse proxy for Zitadel.

I was self-hosting my Zitadel site on port 443 on a subdomain through Nginx (zitadel.mydomain.tld), so it was just a normal HTTPS site. Zitadel seemed to be running fine through the webgui. The Zitadel documentation has you set up an Nginx reverse proxy as follows:

  grpc_pass grpc://127.0.0.1:8080;
  grpc_set_header Host $host:$server_port;

The grpc_set_header line was the culprit for me, breaking my token auth. This was overriding the header that Netbird was sending with the request (documented here). It includes the port, and since I was using the standard HTTPS port, the token verify failed in Zitadel since it didn't want the port included (even though it was the correct port). It's always a string thing. Zitadel's error messages were at least helpful enough in narrowing down the problem.

I removed the $server_port from the grpc_set_header in the Nginx config for Zitadel, and everything started working fine from Netbird. Fingers crossed that this won't create some other fun debugging experience ahead.