[BUG] flytectl cannot validate JWT token

[givanovexpe]

On every flytectl invocation we get an RPC error that points to a JWT verification failure.

could this be related to this issue - https://flyte-org.slack.com/archives/CP2HDHKE1/p1699281995557189

$ flytectl get projects {"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-12-13T14:21:48Z"} {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-12-13T14:21:48Z"} {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-12-13T14:21:48Z"} {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2023-12-13T14:21:48Z"} {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [console] updated. No update handler registered.","ts":"2023-12-13T14:21:48Z"} {"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-12-13T14:21:48Z"} {"json":{"src":"auth_interceptor.go:86"},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-12-13T14:21:51Z"} {"json":{"src":"auth_interceptor.go:91"},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2023-12-13T14:21:51Z"} {"json":{"src":"token_source_provider.go:233"},"level":"info","msg":"retrieved token with expiry 2023-12-13 14:51:52.947038086 +0000 UTC m=+1805.303830171","ts":"2023-12-13T14:21:53Z"} {"json":{"src":"project.go:102"},"level":"debug","msg":"Retrieved 2 projects","ts":"2023-12-13T14:21:54Z"}

== flyteadmin auth setup

userAuth - Okta appAuth - selfAuthServer

server.yaml: | auth: appAuth: selfAuthServer: staticClients: flyte-cli: audience: null
grant_types:

• refresh_token
• authorization_code id: flyte-cli public: true redirect_uris:
• http://localhost:53593/callback
• http://localhost:12345/callback response_types:
• code scopes:
• all
• offline
• access_token flytectl: audience: null grant_types:
• refresh_token
• authorization_code id: flytectl public: true redirect_uris:
• http://localhost:53593/callback
• http://localhost:12345/callback response_types:
• code scopes:
• all
• offline
• access_token flytepropeller: audience: null client_secret: ... grant_types:
• refresh_token
• client_credentials id: flytepropeller public: false redirect_uris:
• http://localhost:3846/callback response_types:
• token scopes:
• all
• offline
• access_token thirdPartyConfig: flyteClient: audience: "" clientId: flytectl redirectUri: http://localhost:53593/callback scopes:
  • offline
  • all authorizedUris:
    • https://
    • http://flyteadmin
    • http://flyteadmin.flyte.svc.cluster.local:80 userAuth: openId: baseUrl: https:///oauth2/default clientId: ... clientSecretFile: ... scopes:
      • profile
      • openid

[dosubot[bot]]

🤖

Hello @givanovexpe! :wave:

I'm Dosu, a friendly bot here to lend a hand with bugs, answer your questions, and guide you on your journey as a contributor while we wait for a human maintainer. Don't hesitate if you need any assistance. Here to help. :robot: :wrench:

Based on the logs you've provided, it seems like the JWT token is not being validated correctly. This could be due to a number of reasons, such as an incorrect or expired token, or a mismatch between the expected audience and the audience claim in the token.

The error message Could not retrieve id token from metadata suggests that the token is not being found in the request metadata. This could be due to the token not being included in the request, or being included in an incorrect format.

The error message Request unauthenticated with IDToken suggests that the request is not being authenticated correctly. This could be due to the token not being valid, or the authentication process not working as expected.

The JWT token validation in the Flyte codebase is handled by the ValidateAccessToken method in the Provider struct. This method parses and validates the token, checks if the token is valid, and verifies the claims against the expected audience. If any of these steps fail, an error is returned.

The findPublicKeyForTokenOrFirst function is used to find the public key for a given token or return the first public key if the token's key is not found. If the KeyID claim is not found or the key corresponding to the KeyID claim is not found in the key set, the function defaults to returning the first public key in the key set.

Based on your server.yaml configuration, it seems like you're using Okta for user authentication and a self-auth server for app authentication. Please ensure that your Okta configuration is correct and that the tokens are being generated and included in the requests correctly.

If you're still facing issues, could you please provide more information about how you're generating and including the tokens in the requests? This would help in diagnosing the issue further. Also, please ensure that your Flyte setup is up-to-date with the latest version, Flyte v1.10.6 milestone release.

For more information, you can refer to the Flyte Admin Auth documentation and the Flyte Admin codebase.

I hope this helps! If you have any more questions, feel free to ask.

-Dosu

Sources

• flyteadmin/auth/authzserver/provider.go
• flyteadmin/auth/authzserver/provider_test.go

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[a4649]

Did you manage to solve it?