HttpSenderEventListenerProviderFactory, error: java.util.ConcurrentModificationException, KC 25.0.2 or 25.0.4

[adrian-moisa]

As soon as I enable the http-sender in realm events I keep getting this error on more than half of the endpoints of my app. It seems to happen on token introspect, or somewhere after. It does not have a clear rule for triggering. Some EPs of my REST api work some not. It happens at random. Feels like a race condition.

I worked with http-sender with a much older version KC 22. And there I did not have this issue. Problem is that I need to upgrade to KC 24 minimum to get the benefit of dynamic profile fields. If I upgrade I stumble into this erorr. It keeps occurring at random.

2024-08-31 20:52:35,606 INFO  [io.quarkus.http.access-log] (executor-thread-2) 192.168.65.1 - - [31/Aug/2024:20:52:35 +0000] "POST /realms/myapp/protocol/openid-connect/token/introspect HTTP/1.1" 500 113
2024-08-31 20:52:35,624 INFO  [io.phasetwo.keycloak.events.HttpSenderEventListenerProviderFactory] (executor-thread-2) Configuring io.phasetwo.keycloak.events.HttpSenderEventListenerProvider with [targetUri=http://host.docker.internal:13100/v1/users/events]
2024-08-31 20:52:35,625 INFO  [io.phasetwo.keycloak.events.HttpSenderEventListenerProviderFactory] (executor-thread-1) Configuring io.phasetwo.keycloak.events.HttpSenderEventListenerProvider with [targetUri=http://host.docker.internal:13100/v1/users/events]
2024-08-31 20:52:35,629 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-2) Uncaught server error: java.util.ConcurrentModificationException
        at java.base/java.util.HashMap$Values.forEach(HashMap.java:1076)
        at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:385)
        at org.keycloak.models.KeycloakBeanProducer_ProducerMethod_getKeycloakSession_XoSEUTXOsE3bpqXlGMAykCiECUM_ClientProxy.close(Unknown Source)
        at org.keycloak.quarkus.runtime.transaction.TransactionalSessionHandler.close(TransactionalSessionHandler.java:60)
        at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.closeSession(CloseSessionFilter.java:67)
        at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.filter(CloseSessionFilter.java:63)
        at org.jboss.resteasy.reactive.server.handlers.ResourceResponseFilterHandler.handle(ResourceResponseFilterHandler.java:25)
        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:150)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Edit: I tried version 24.0.5 - This one seems ok, I don't see the error here. All stable. Can work with 24, but would be nice to be able to upgrade sooner than later. Cheers!

[xgp]

@adrian-moisa Can you post the exact Keycloak and library versions you are seeing those errors, and a complete step-by-step instructions on how to reproduce so that we can debug?

[adrian-moisa]

I'm using the official p2 kc image. I tried several versions from the 25 range, including latest. they all seem to have the same problem. Latest 24 (24.0.5) does not have it. As for setup. I don't think I'm doing anything special. Just a plain old authorisation flow on the web, using flutter web and some token introspection from the server side using gocloak. I'm having trouble identifying what makes my setup different/special to create the bug conditions. If it helps at all maybe we can do a screen share one day. potentially you can spot smth.

I'm running on a mac M1, KC is in docker and the go apis are on local started from vscode. I'm not having an excessive amount of endpoints running. 2-4 go in parallel at once depending on the situation.

[xgp]

I just tried a few cases like that on the most recent version, and didn't see the issue.

Please do a video of everything that is happening so we can try to reproduce.

[mshivanna]

Same behavior on KC22. We tried to write to a HTTP endpoint. We use 0.32 version of events jar.

[mshivanna]

Does not happen with 0.28. version of the jar

[xgp]

@mshivanna Please test with the https://github.com/p2-inc/keycloak-events/tree/xgp/simplehttp-legacy branch. I think we've solved this in #80 and can use additional testing. Also, This version will only work with Keycloak 25.

[mshivanna]

I did build the jar from that but got no class exception. I can get you the exact error. but our keycloak version is old. kc22 so my testing will be useless. We are ok to use v0.28 of the jar.

[xgp]

Fixed by https://github.com/p2-inc/keycloak-events/pull/80