"Error parsing token: unable to find appropriate key" fixed with restarting management service

[mohamed-essam]

Describe the problem

Randomly every few days, netbird up fails with Error: waiting sso login failed with: rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key, this is fixed by restarting management service

To Reproduce

Unknown

Expected behavior

Connection to be established normally

Are you using NetBird Cloud?

Self-hosted.

NetBird version

0.28.4

NetBird status -d output:

N/A

Screenshots

N/A

Additional context

Management server logs:

management-1  | 2024-07-14T07:18:21Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
management-1  | 2024-07-14T07:18:21Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue
management-1  | 2024-07-14T07:18:22Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
management-1  | 2024-07-14T07:18:22Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue
management-1  | 2024-07-14T07:18:22Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
management-1  | 2024-07-14T07:18:22Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue

iDP Used: Google Workspace

netbird up -F -l debug output:

2024-07-14T10:26:02+03:00 DEBG client/internal/login.go:93: connecting to the Management service <REDACTED>
2024-07-14T10:26:02+03:00 DEBG client/internal/login.go:63: connected to the Management service <REDACTED>
2024-07-14T10:26:03+03:00 DEBG client/internal/login.go:72: peer registration required
2024-07-14T10:26:03+03:00 DEBG client/internal/login.go:122: sending peer registration request to Management Service
2024-07-14T10:26:04+03:00 ERRO client/internal/login.go:126: failed registering peer rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key,00000000-0000-0000-0000-000000000000
Error: foreground login failed: login failed: rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key

[Zwordi]

Hi,

I’ m having the same situation and same behavior with latest and G.Workspace.

Thanks for creating the issue.

[braginini]

hey @mohamed-essam and @Zwordi Does this happen on older NetBird versions? Could you please share the generated JWT token contents jwt.io through Slack?

[mohamed-essam]

Hello @braginini,

Do you mean client or management versions? As this installation is used by multiple other people within my organization I will be unable to downgrade the server version to test for extended amounts of time.

As for the generated JWT token contents, does that appear in debug logs? Or do I need to do something specific to get that once the error occurs?

I took a quick look in the code and I think the root cause may be a failure in updating the JSONWebKey in https://github.com/netbirdio/netbird/blob/main/management/server/jwtclaims/jwtValidator.go#L108 , I turned on debug logging yesterday and waiting for the issue to occur again to be able to share the debug logs around the time the issue starts.

On a separate topic I believe that line of logging should definitely be a Warn or Error not Debug.

[mohamed-essam]

Some extra information I forgot to include: This issue is most likely server-side as it caused all SSO clients to be unable to connect (my own client and 4 other personnel were unable to authenticate)

[mohamed-essam]

I found that the config generated by the setup script has HttpConfig.IdpSignKeyRefreshEnabled set to false, changed it to true manually, and will check if it works and report back

Side note: the issue occurred again today, it seems to be occurring almost weekly

[mohamed-essam]

This week no issue occurred, the issue seems to be the setup script disables refreshing idp keys for Google workspace when it should be enabled

[a8uhnf]

I found that the config generated by the setup script has HttpConfig.IdpSignKeyRefreshEnabled set to false, changed it to true manually, and will check if it works and report back

Side note: the issue occurred again today, it seems to be occurring almost weekly

thanks a lot. seems working for me. IMO, for SSO it should be default behaviour