jwks Error-IstiodFailedToFetchJwksUri

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

I have a request authentication deployed in my namespace

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: "example-jwt"
spec:
  selector:
    matchLabels:
      app: my-app
  jwtRules:
  - issuer: "https://my-outside-cluster-host.com/oidc"
    outputPayloadToHeader: "x-jwt-payload"

All my Auth requests fail with Jwks doesn't have key to match kid or alg from Jwt. error

Upon inspection of logs of istio-proxy inside my-app pod I see weird/old values of jwks

Note: example.com is not what t is - it is an outside cluster machine

{
  "level": "debug",
  "time": "2024-09-24T10:29:40.948252Z",
  "scope": "envoy config",
  "msg": "    config: {\"@type\":\"type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication\",\"providers\":{\"origins-0\":{\"issuer\":\"https://my-outside-cluster-host.com/cf/oidc\",\"local_jwks\":{\"inline_string\":\"{\\\"keys\\\":[ {\\\"e\\\":\\\"AQAB\\\",\\\"kid\\\":\\\"abc\\\",\\\"kty\\\":\\\"RSA\\\",\\\"n\\\":\\\"RXJyb3ItSXN0aW9kRmFpbGVkVG9GZXRjaEp3a3NVcmkt\\\"}]}\"},\"forward_payload_header\":\"x-jwt-payload\",\"payload_in_metadata\":\"https://my-outside-cluster-host.com/oidc\"}},\"rules\":[{\"match\":{\"prefix\":\"/\"},\"requires\":{\"requires_any\":{\"requirements\":[{\"provider_name\":\"origins-0\"},{\"allow_missing\":{}}]}}}],\"bypass_cors_preflight\":true}",
  "caller": "external/envoy/source/common/http/filter_chain_helper.h:138",
  "thread": 17
}

Notice the kid: abc and n: RXJyb3ItSXN0aW9kRmFpbGVkVG9GZXRjaEp3a3NVcmkt

Further mystery

echo RXJyb3ItSXN0aW9kRmFpbGVkVG9GZXRjaEp3a3NVcmkt | base64 -d

Error-IstiodFailedToFetchJwksUri- wow

So I have following bugs to report:

1) Can istiod retrieve from my-outside-cluster-host.com ? I do have a ServiceEntry and Destination rule against this host. So why istiod not able to retieve from it.

2) Should the errors be obfustcated with bae64 ?

3) Why the deployment of such RequestAUthentication does not fail if it can't retrieve the jwks ?

At this time I am interested in getting solution/workaround for 1) fixed. Wondering if anybody has idea.

Version

1.20.2
Mote - I do not have access to list resources under istio-system) but from the logs

Additional Information

No response

istio / istio