Container marked as initialized before admin password is populated

[ZIRAKrezovic]

Describe the bug

I have a problem where container is marked as ready and KeycloakClient starts calling it before admin user has been created.

This is quite random and (probably) depends on system load. The current probe looks for HTTP port, which becomes available before the admin is initialized, resulting in 401 when calling any ADMIN API.

Version

3.3.0

Expected behavior

I would like a guarantee that I can use admin user immediately after "start" command returns.

Actual behavior

5 out of 10 times, calling admin API with provided admin user results in 401 as admin gets created a bit later (several milliseconds to two seconds difference)

How to Reproduce?

A simplest reproducer was to start two Keycloak container instances. I have multiple @SpringBootTest-s in my app, each runs in different Java container so each of them spawns their own Keycloak container. I can try and get a reproducer in plain java if you need it.

Relevant log output

2024-04-19 13:11:18,085 INFO  [io.quarkus] (main) Profile dev activated.
2024-04-19 13:11:18,085 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, keycloak, logging-gelf, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, vertx]
2024-04-19 13:11:18,578 WARN  [org.keycloak.events] (executor-thread-2) type="LOGIN_ERROR", realmId="bc1830f9-43e4-4ea2-9799-70a1ab02e8b4", clientId="admin-cli", userId="null", ipAddress="172.29.0.1", error="user_not_found", auth_method="openid-connect", grant_type="password", client_auth_method="client-secret", username="admin"
2024-04-19 13:11:18,810 INFO  [org.keycloak.services] (main) KC-SERVICES0009: Added user 'admin' to realm 'master'
2024-04-19 13:11:18,816 WARN  [org.keycloak.quarkus.runtime.KeycloakMain] (main) Running the server in development mode. DO NOT use this configuration in production.

Anything else?

No response

[dasniko]

Thanks for reporting the issue. I've also already discovered this behavior, but haven't found a proper solution. I've experimented with the various health endpoints, but even the started endpoint does not work reliable. Currently I have not idea how to solve this and have still to explore other possibilities. if you have any ideas, they are more than appreciated!

[ZIRAKrezovic]

I have filled a request in Keycloak repo asking for a (semi) official implementation for what you described.

[dasniko]

@ZIRAKrezovic Pls. try #136 - this should solve the issue. Can you confirm?

[ZIRAKrezovic]

Hi, it looks like it may fix the issue for the described use-case. But I may have seen some times where the log message for dev mode is logged before admin user is created - again quite hard to reproduce.

[ZIRAKrezovic]

The last solution I'll try if this one fails will be to ship a pre-initialized H2 database with admin user and mount it as a file - but then again this is out of scope of this project.

[dasniko]

But I may have seen some times where the log message for dev mode is logged before admin user is created

TBH, I haven't seen this ever. 🤷‍♂️

[dasniko]

But I may have seen some times where the log message for dev mode is logged before admin user is created

TBH, I haven't seen this ever. 🤷‍♂️

[lmachacek]

We had the same problem and this waiting strategy fixed that, thanks :+1:

[ZIRAKrezovic]

And to debunk myself, I have started 10 keycloak containers on the same machine and all 10 of them were initialized correctly - with the log message about dev mode being last

for i in 0 1 2 3 4 5 6 7 8 9 ; do docker run -d --rm --name keycloak-$i keycloak/keycloak:23.0.7 start-dev ; done

Thanks for the promt fix @dasniko.