Keycloak HA Infinispan and Istio: jgroups should not be exposed

[leahneukirchen]

In case other people try to do this: if you want to use HA Keycloak and run Istio, you need to remove containerPort 7600 jgroups, because Istio will not permit direct pod to pod communication of exposed ports.

Just removing this pod makes dns.DNS_PING pick up the other pods fine.

[ivans3]

+1 ,tho we may still need to use the JDBC based discovery method, as i was seeing this in the logs for 1/3 of the pods:

00:27:37,045 WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,uaa-keycloak-1) JGRP000032: uaa-keycloak-1: no physical address for 248840c6-18f3-d786-9466-a4e3ed2e0df6, dropping message

[zodiac1214]

In case other people try to do this: if you want to use HA Keycloak and run Istio, you need to remove containerPort 7600 jgroups, because Istio will not permit direct pod to pod communication of exposed ports.

Just removing this pod makes dns.DNS_PING pick up the other pods fine.

Still new to helm chart. Is there way to remove 7600 without fork the whole keycloak helm chart? I use this chart as my dependency :(

[fibbers]

I was also figuring out how to get clustering to work with Istio and stumbled upon this issue (thanks @leahneukirchen). Though, if I understood correctly you need to (manually) modify the service and remove the jgroups part.

As an alternative, you can also exclude this port from Istio. Reading https://istio.io/docs/reference/config/annotations/ what works for me is:

keycloak:
  ...
  podAnnotations:
    traffic.sidecar.istio.io/excludeInboundPorts: '7600'

[leahneukirchen]

We rewrite the Helm charts anyway on the fly with Jsonnet, but this is a good Helm-only solution. Thanks!

[brantlk]

We've got istio with mTLS enabled and I also needed to set traffic.sidecar.istio.io/excludeOutboundPorts: '7600'

Otherwise keycloak failed with an error about mismatching cookie.

[unguiculus]

@leahneukirchen Please have a look at #220 which adds an option to disable JGroups ports. I tested it with two replicas. JGroups still keeps working correctly.

[unguiculus]

Fixed in #220

[TDesai10]

I believe I am running into the same issue as well - I see the pods up but cannot login to the admin console on keycloak and initially the logs stated that it could not discover other pods.

So, i added the podAnnotation but now I see the below error, Can someone please help me with this? keycloak 15:06:16,868 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7600]-2,null,null) JGRP000006: failed accepting connection from peer: java.net.SocketException: 10.208.120.46:7600: BaseServer.TcpConnection.readPeerAddress(): cookie sent by /10.208.124.147:43620 does not match own cookie; terminating connection

[philippebailer]

I was having issues getting KC to work in istio and this was the only somewhat useful issue that would come up in google search so I'm making this comment here for others sanity maybe

I'm not directly using this chart but I am looking through it as a guide and what I found was

1. istio doesn't like it when you expose the jgroup port which is fixed by the 220 PR so make sure to not do that
2. you DO NEED the headless service type: ClusterIP for the DNS_PING which is the --cache-stack=kubernetes default right now so set that up additionally to your ingress service
3. The default port for jgroup is currently 7800 not 7600 as stated above so if you have issues check the port thats getting blocked
4. You need to exclude both IN and OUT bound ports for infinispan to do its thing in istio

     annotations:
       traffic.sidecar.istio.io/excludeInboundPorts: "7800"
       traffic.sidecar.istio.io/excludeOutboundPorts: "7800"

• I wasnt able to get istio working without the above
• Trying outboundTrafficPolicy mode allow_any failed to work during testing
  • I was able to connect to kc on the desired port from a busybox container
  • but kc <> kc would still fail to set up the infinispan cluster

SIDENOTE: KUBE_PING is a different solution than DNS_PING that from my understanding requires the rbac policies youll find in this chart but the default DNS_PING does not need that it just needs the headless service to get the ip list so if your not using the default Ive no idea if this will help you

If anyone stopping by in the future has a better solution or more info please share!

[sirishkumar]

@philippebailer

Its also required to whitelist port 57800 in addition to 7800 for Failure detection. Atleast in Keycloak version 24.0.5 and earlier, we didnt see any visible impact of blocking this port for longtime

https://github.com/keycloak/keycloak/issues/20563 https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/running/timeout_tunning

Otherwise you may see logs similar to below. If port 57800 is not whitelisted looks like infinispan will incrementally try for other ports starting from 57801

keycloak-2 istio-proxy [2024-09-17T13:21:03.990Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 10.175.69.83:57800 10.175.85.132:38599 - -