Signature validation failed. Unable to match key

[MaxThom]

Describe the bug

Hi, thank you for this amazing solution.

My solution is deployed on k8s with Azure. I have an ingress with lets encrypt. I've used the templates so I got admin, adminapi and identity.

I often have this error message in my logs and I have to restart the pod. Any idea what could cause the issue?

Thank you !

To Reproduce

Deploy on azure k8s with ingress and letsencrypt for https

Relevant parts of the log file

[13:41:39 ERR] Exception occurred while processing message.
Microsoft.IdentityModel.Tokens.SecurityTokenSignatureKeyNotFoundException: IDX10501: Signature validation failed. Unable to match key:
kid: 'System.String'.
Exceptions caught:
 'System.Text.StringBuilder'.
token: 'System.IdentityModel.Tokens.Jwt.JwtSecurityToken'.
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateSignature(String token, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateToken(String token, TokenValidationParameters validationParameters, SecurityToken& validatedToken)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.ValidateToken(String idToken, AuthenticationProperties properties, TokenValidationParameters validationParameters, JwtSecurityToken& jwt)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleRemoteAuthenticateAsync()
[13:41:39 ERR] An unhandled exception has occurred while executing the request.
System.Exception: An error was encountered while handling the remote login.
 ---> Microsoft.IdentityModel.Tokens.SecurityTokenSignatureKeyNotFoundException: IDX10501: Signature validation failed. Unable to match key:
kid: 'System.String'.
Exceptions caught:
 'System.Text.StringBuilder'.
token: 'System.IdentityModel.Tokens.Jwt.JwtSecurityToken'.
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateSignature(String token, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateToken(String token, TokenValidationParameters validationParameters, SecurityToken& validatedToken)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.ValidateToken(String idToken, AuthenticationProperties properties, TokenValidationParameters validationParameters, JwtSecurityToken& jwt)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleRemoteAuthenticateAsync()
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler`1.HandleRequestAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

[ChristianAVisma]

We are seeing the same issue, but with a different hosting env. We do however experience the same issue, and we think the loadbalancer is the issue. We haven't been able to confirm this 100% though.

[MaxThom]

ok, for my loadbalancer I used nginx with sticky session. I also have 2 replicas for the identity, admin and api. Here is my yaml for the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: senseai-ingress
  namespace: frontend
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "affinity"
    nginx.ingress.kubernetes.io/session-cookie-expires: "14400"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "14400"
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
    nginx.ingress.kubernetes.io/proxy-buffer-size: 32k
    nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
spec:
  tls:
  - hosts:
    - identity-admin-api-qa.senseai.com
    - identity-admin-qa.senseai.com
    - identity-server-qa.senseai.com
    secretName: tls-secret
  rules:
  - host: identity-server-qa.senseai.com
    http:
      paths:
      - path: /(.*)
        pathType: "Prefix"
        backend:
          service:
            name: identity-server-service
            port: 
              number: 80
  - host: identity-admin-qa.senseai.com
    http:
      paths:
      - path: /(.*)
        pathType: "Prefix"
        backend:
          service:
            name: identity-admin-service
            port: 
              number: 80
  - host: identity-admin-api-qa.senseai.com
    http:
      paths:
      - path: /(.*)
        pathType: "Prefix"
        backend:
          service:
            name: identity-admin-api-service
            port: 
              number: 80

Hope it helps! Gl :)

[xiongtec]

@MaxThom curious if you find the solution.

[apetrut]

@MaxThom we are trying to deploy sts, admin and admin-api services in K8S in Azure but we are not using Nginx (we use Azure App Gateway).

Only the STS service is up and running and admin and admin-api are not. The health check is failing for admin and admin api and we got these errors:

@skoruba Can you confirm there is a dependency between STS, Admin and Admin Api services and Nginx? Is there a way to deploy these services without nginx?

The Ids health check inside admin service is basically trying to reach the discovery document from sts service and it fails. However, if I try to load the discovery document from my browser it works. Hence I suspect it has something to do with communication between the admin service and sts service inside the K8S cluster.