Keycloak Cannot Cluster with MTLS Strict Peerauthentication, Admin Console Too Many Redirects

[trossb]

Before reporting an issue

• [X] I have searched existing issues
• [X] I have reproduced the issue with the latest release

Area

authentication

Describe the bug

We are having trouble running Keycloak in AWS EKS with Istio sidecars attached and peerauthentication set to strict. We can send postman requests to receive auth tokens, but we cannot log into the admin console. When a login is attempted, we get a "Too many redirects" error from chrome. It appears that our HPA configuration is no longer working and this is the cause of this. The nodes in the cluster cannot find each other when Keycloak starts up. If we set the replicas down to 1, we are able to login to the console, or if we set the peerauthentication to permissive, we do not have this issue.

Error when Keycloak starts and cannot form a cluster with all the replicas: 2023-03-24 18:31:10,963 WARN [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-0-39769: too many JOIN attempts (10): becoming singleton

Version

19.0.3

Expected behavior

Attaching Istio sidecars and strict peerauthentication should allow login to admin console and the Keycloak replicas should be clustered.

Actual behavior

Cannot login to the admin console.

How to Reproduce?

Set keycloak deployment annotations: sidecar.istio.io/inject: "true", sidecar.istio.io/rewriteAppHTTPProbers: "true"

Set ingresss annotations: nginx.ingress.kubernetes.io/service-upstream: "true" nginx.ingress.kubernetes.io/upstream-vhost: "keycloak.my-namespace.svc.cluster.local"

Configure peerauthentication: metadata: name: default namespace: my-namespace spec: mtls: mode: STRICT

Anything else?

We also found this in our research https://repo1.dso.mil/big-bang/bigbang/-/issues/651 apparently Keycloak is not a fan of Istio.

[sschu]

If this is something with your specific network environment like your usage of istio, I don't think that is a bug in Keycloak as this is not officially supported.

[srivasud]

Hi @sschu If you notice the error message "Error when Keycloak starts and cannot form a cluster with all the replicas". My question is, Keycloak running in HPA (Horizontal Pod Autoscaler) cluster with Istio and strict mTLS peerauthentication mode is supported by Keycloak ? If we keep the mTLS peerAuthentication to 'PERMISSIVE' and keeping the other properties the same as before (i.e HPA with istio) we don't see this error and it works fine. Please let us know.

[sschu]

There is no official support for this setup. So maybe it works, maybe it doesn't nobody has verified this. I don't know much about Istio so I cannot tell you what a correct setup would look like. What I do know is that Keycloak builds a cluster and nodes need to communicate with each other. This might definitely be affected if you add Istio and mtls to the network setup.

[mposolda]

@sschu Thanks for replying to this!

+1 that this is not a bug as Keycloak doesn't claim anywhere to setup the environment like this. It may need to be new task/feature (if something) to support something like that.