STANDALONE CLUSTER UNABLE TO SYNCHRONIZE IMMEDIATELY

[hakimnorizman-work]

Hi, I have setup 2 (two) keycloak machine which is kecloak01, and keycloak02, both of this is connected to another standalone MariaDB server. In testing environment, I created a realm in the primary keycloak (keycloak01) which is TEST01, but in secondary server (keycloak02) does not show TEST01 realm, note that both keycloak is turned on. But when I created another realm in keycloak02 which is TEST02, then the TEST01 realm is visible once after I created the new realm in keycloak02. This is not a problem if the keycloak01 is down and up again, it will get the latest realm/data from the database. Seems that it need to reconnect to the database server again if want to sync the data. My question is, is there a way so that both keycloak server will get the data sync all the time?

[ivangfr]

Hi @hakimnorizman-work I was able to reproduce your case.

Basically, what I did was to have running two Keycloak instances connected to MariaDB.

Then, I accessed the Keycloak-1 UI and created a realm called Test. After that, going to the Keycloak-2 UI, without refreshing the page, the realm Test is not shown. However, it's shown just when I refresh the Keycloak-2 UI page. By refreshing the page, the instance is going to DB, as you said.

The same happens the other way around, i.e, creating the realm in Keycloak-2 UI and checking in Keycloak-1 UI.

In order to have the realm sync real-time, without refreshing the page (going to DB), I believe in Keycloak, at the moment, there isn't any implementation (such as WebSocket) that supports it.

[hakimnorizman-work]

Such a pity if no WebSocket implementation. But do you able to synchronize it by just refreshing the page? because I was not able to do so. I have to create another realm then the other created realm appears. Is it because of cookies? This is my setup https://github.com/hakimnorizman-work/keycloak-jdbc-ivanfranchin Btw, thank you for the tips.

[ivangfr]

In my test, I've followed this steps described in README of keycloak-clustered.

In fact, it's different from yours because MariaDB and the two Keycloak instances are in the same network, called keycloak-net.

In the README, I have also a section called _Running a Keycloak Cluster using JDBCPING in Virtual Machines where I use Vagrant to create 3 virtual machines exclusively to each service, i.e, one VM to Database, another VM to Keycloak1 and, finally, another VM to Keycloak2. Maybe, this would be similar to what you are trying to do.

Btw, after reading the README of your project, I didn't get how you run MariaDB and the two Keycloak instances, and connect all of them. Could you update the README, describing step-by-step, how to put your project up-and-running? This way, I could try to reproduce in my machine.

[hakimnorizman-work]

Sure.. I have updated the steps.. if you are not sure.. you can ask me. Btw, I'm really thankful for your help. :)

[ivangfr]

Hi @hakimnorizman-work thanks for the update in README.

Unfortunately, I won't be able to try your project, as you are using VMware Workstation and, in my Mac I don't have it and, it looks like, VMware Workstation is supported just in Windows and Linux.

Anyway, I've tried again the realm creation, however, this time I used Virtual Machines (as describe at _"Running a Keycloak Cluster using JDBCPING in Virtual Machines".

I've found an issue in the Vagrant file, but, after fixing it, the realm creation worked as excepted. In order to have the new realm in the another instance, I need to just refresh the page.

But, it's weird what is happening with your setup. Usually, the refresh of the page should be enough to have the latest realms.

[hakimnorizman-work]

Hi @ivangfr Thanks for the reply After I read your vagrant file, I found the line below mentioning for open port 8080, 7800, and 8443. "KEYCLOAK_CLUSTERED_CONTAINER_ARGS = "-p 8080:8080 -p 7800:7800 -p 8443:8443"

Is there a significant in these ports? I'm just afraid I'm missing something in the configurations. Have you read my setup files? Is there any problems with it? Thank you in advance.

Btw, I did found an issue where there were only one populated IP in JGROUPSPING database https://keycloak.discourse.group/t/use-of-jdbc-ping-with-keycloak-17-quarkus-distro/13571/29?u=hakimnorizman

[ivangfr]

8080 is http, 8443 is https, 7800 is tcp. To my Vagrantfile, I believe just the port 8080 should be exposed as, later, I am forwarding it to the main host in this line.

[DAHAG-ArisNourbakhsh]

"realms" are, per default, locally cached on each Keycloak Node for up to an hour. When using Inifnispan each keycloak node will post invalidation keys messages to the cluster to notify that they need to re-fetch locally stored infomation (like realms) from the DB.

I strongly suspect that your clustering setup is not working at all and that the nodes are not able to form a cluster and thus unable to exchange the invalidation messages.

@hakimnorizman-work

I have two questions for you:

1) Could you demonstrate/show that the containers are indeed using the 192.168.1.2 and 192.168.1.3 IP Address? Because from what I can see from the steps you outlined none of the commands or docker-compose configuration would do that.

• docker inspect the containers to show which networks they are part of
• and attach into the container and make SURE that the interfaces have the expected IP Addresses

2) Could you show the logs during the startup of both keycloak instances just to see if they report a successful cluster formation?

Maybe as a reference on how I solved getting everything to work with docker-compose (and docker swarm):

• single compose file with multiple replicas
• defined extra infinispan network to disambiguate between multiple networks (traefik and the internal swarm network)

I'm not sure if this is 100% the way to go but I thought it was neat to get this to work with docker-swarm and being able to use replicas properly. (Also this is a very abridged example that I use for local testing sometimes)

version: '3'

volumes:
  postgres_data: 

services:
  postgres:
    image: [private postgres image]
    volumes: 
      - postgres_data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD: password
    ports:
      - 5432:5432
    networks:
      - db
  keycloak:
    image: [Custom Image]
    deploy:
      replicas: 3
    environment:
      KC_HOSTNAME: localhost:8080
      KEYCLOAK_ADMIN_PASSWORD: admin
      KC_DB: postgres
      KC_DB_URL_HOST: postgres
      KC_DB_URL_PORT: 5432
      KC_DB_URL_DATABASE: keycloak
      KC_DB_URL_PROPERTIES: ?prepareThreshold=0
      KC_DB_USERNAME: keycloak 
      KC_DB_PASSWORD: password 
      KC_DB_SCHEMA: public
      INFINISPAN_PROBE: 172.212.0.1
      INFINISPAN_BIND_ADDRESS: match-address:172.212.*
    entrypoint:
      - "/opt/keycloak/bin/custom-entry.sh"
      - "start-dev"
    depends_on:
      - postgres
    networks:
      - keycloak-infinispan
      - db

networks:
  db:
  keycloak-infinispan:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.212.0.0/16

custom-entry.sh (instead of kc.sh)

#!/bin/sh

echo "Determining own Ip Address for infinispan by searching for next best interface that would route $INFINISPAN_PROBE"

ROUTE_OUTPUT=$(ip route get "$INFINISPAN_PROBE")
echo "route to $ROUTE_OUTPUT"
GREP_RESULT=$(echo "$ROUTE_OUTPUT" | grep -oP "src (172\.21\d\.\d*.\d*)")
echo "Grep $GREP_RESULT"
OWN_IP_ADDRESS=$(echo "$GREP_RESULT" | cut -d" " -f2)

echo "Own Ip Address is: $OWN_IP_ADDRESS"

export OWN_IP_ADDRESS=$OWN_IP_ADDRESS

echo "attempting to launch keycloak"

/opt/keycloak/bin/kc.sh "$@"

Also changed the cache-ispn-jdbc-ping slightly

<TCP external_addr="${env.OWN_IP_ADDRESS:127.1.1.1}" bind_addr="${env.INFINISPAN_BIND_ADDRESS:127.1.1.0}"/>

our dockerfile

FROM quay.io/keycloak/keycloak:19.0.2 as builder

ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENV KC_DB=postgres
ENV KC_CACHE_CONFIG_FILE=cache-ispn-jdbc-ping.xml
COPY deployments/. /opt/keycloak/providers/
COPY conf/. /opt/keycloak/conf/
RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:19.0.2
USER root
RUN microdnf update -y && microdnf install iproute
COPY custom-entry.sh /opt/keycloak/bin/custom-entry.sh
RUN chmod 777 /opt/keycloak/bin/custom-entry.sh
RUN chown keycloak /opt/keycloak/bin/custom-entry.sh
USER 1000

COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
WORKDIR /opt/keycloak
COPY deployments/. ./providers/ 
COPY conf/. ./conf/
HEALTHCHECK --start-period=120s CMD curl --fail http://localhost:8080/health || exit 1

ENTRYPOINT ["/opt/keycloak/bin/custom-entry.sh", "start", "--optimized"]