jmederer
commented
11 months ago Hi @xgp,
it looks like cassandra is not completely up and running. Can you please check logs of cassandra and when its up then start keycloak.
Closed xgp closed 10 months ago
jmederer
commented
11 months ago Hi @xgp,
it looks like cassandra is not completely up and running. Can you please check logs of cassandra and when its up then start keycloak.
xgp
commented
11 months ago Hi @jmederer. Thanks for looking at this.
it looks like cassandra is not completely up and running.
I suspected that and waited for 10 minutes. Still the same result. Cassandra settles (no more logs):
INFO [main] 2023-12-15 15:33:05,822 PipelineConfigurator.java:131 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO [main] 2023-12-15 15:33:05,826 CassandraDaemon.java:738 - Startup completeand I'm able to use cqlsh to look around:
root@cf0981aebd6a:/# cqlsh
WARNING: cqlsh was built against 5.0-beta1, but this server is 5.0. All features may not work!
Connected to Test Cluster at 127.0.0.1:9042
[cqlsh 6.2.0 | Cassandra 5.0-beta1 | CQL spec 3.4.7 | Native protocol v5]
Use HELP for help.
cqlsh> describe cluster
Cluster: Test Cluster
Partitioner: Murmur3Partitioner
Snitch: DynamicEndpointSnitchI've validated the networks between Keycloak and Cassandra. Do you know of anything else I should test? Thanks for the help.
xgp
commented
11 months ago A few updates, but same result in all cases.
xgp
commented
11 months ago Is there a docker-compose or another known good template to use to get a working version?
opdt
commented
11 months ago While trying to replicate your problems, we found a few other issues when trying to run the extension by copying to the providers-folder as well (We build our own distribution so we didn't find this before).
We are still not sure what your exact problem is. Could you elaborate further on your specific setup (like OS used)? After we fix aforementioned issues, we will provide an example Dockerfile and docker-compose.yaml.
xgp
commented
11 months ago Hi @opdt Thanks for looking again. Yes, I noticed that there was a problem with putting the extension in the providers dir. I solved it by using the maven shade plugin to put all the deps in the same jar.
diff --git a/core/pom.xml b/core/pom.xml
index c2f4b9d..9706c51 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -76,4 +76,33 @@
</dependencies>
+ <build>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>3.2.4</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/MANIFEST.MF</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ Regarding the specific setup. I'm running on Ubuntu on amd64 with Keycloak and Cassandra in docker. My docker-compose file currently looks like this:
version: '3'
volumes:
caddy_data:
driver: local
services:
cassandra:
image: cassandra:4
ports:
- 9042:9042
environment:
MAX_HEAP_SIZE: 256M
HEAP_NEWSIZE: 128M
CASSANDRA_DC: datacenter1
CASSANDRA_CLUSER_NAME: cassandra
keycloak:
image: quay.io/keycloak/keycloak:22.0.5
volumes:
- ${PWD}/core/target/keycloak-cassandra-extension-1.3.2-22.0.1-SNAPSHOT.jar:/opt/keycloak/providers/keycloak-cassandra-extension.jar
- ${PWD}/delay-kc.sh:/opt/keycloak/bin/delay-kc.sh
environment:
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: admin
KEYCLOAK_ADMIN: admin
KEYCLOAK_ADMIN_PASSWORD: admin
KC_HOSTNAME_STRICT: 'false'
KC_HTTP_ENABLED: 'true'
KC_PROXY: 'edge'
KC_LOG_LEVEL: DEBUG
KC_SPI_DATASTORE_PROVIDER: cassandra-map
KC_SPI_DATASTORE_CASSANDRA_MAP_CACHE_MODE: false
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_PORT: 9042
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_CONTACT_POINTS: cassandra
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_USERNAME: cassandra
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_PASSWORD: cassandra
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_LOCAL_DATACENTER: datacenter1
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_KEYSPACE: test
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_REPLICATION_FACTOR: 2
ports:
- 8080:8080
depends_on:
- cassandra
entrypoint: /opt/keycloak/bin/delay-kc.sh --verbose start
caddy:
image: caddy:2.4.6-alpine
restart: unless-stopped
command: caddy reverse-proxy --from https://localhost:443 --to http://keycloak:8080
ports:
- 80:80
- 443:443
volumes:
- caddy_data:/data
depends_on:
- keycloakand that delay-kc.sh file just sleeps before starting Keycloak to give Cassandra some time to settle.
#!/bin/bash
echo "Sleeping for 5 minutes!"
sleep 300
# run the keycloak entrypoint with the given params
RUN_OPTS="$@"
/opt/keycloak/bin/kc.sh $RUN_OPTSLet me know if you need any more information, or would like me to test with any changes. I really appreciate your help in getting this working, and I'm excited to help out once I can get a working version I can test/modify.
xgp
commented
11 months ago I was able to spend some more time on this. My guess is you're testing/running on a real Cassandra cluster, rather than a single node for testing. I updated the following to get it to start:
KC_SPI_CASSANDRA_CONNECTION_DEFAULT_REPLICATION_FACTOR=1. This solved the "No node was available" error.DefaultCassandraConnectionProviderFactory.java file Database database = new Database(cqlSession, mgConfig).setConsistencyLevel(ConsistencyLevel.ONE). This solved a similar error about only having one node for ALL or QUORUM.However, after that, any attempt to submit any web requests (logins) yields a strange error after startup. My guess is there is probably some conflict in the jars that got included that is producing the conflict. Need to debug that more. Any insight on how you are bundling things would be helpful.
2023-12-22 07:45:37,051 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-3) Uncaught server error: java.io.IOException: java.nio.channels.ClosedChannelException
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.readBlocking(VertxInputStream.java:256)
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.readIntoBuffer(VertxInputStream.java:123)
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:85)
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:73)
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:64)
at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler$CapturingInputStream.read(FormBodyHandler.java:131)
at java.base/java.io.InputStream.read(InputStream.java:284)
at java.base/java.io.InputStream.read(InputStream.java:218)
at org.jboss.resteasy.reactive.server.core.multipart.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:235)
at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler.handle(FormBodyHandler.java:94)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:150)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:145)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.nio.channels.ClosedChannelException
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.<init>(VertxInputStream.java:184)
at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.<init>(VertxInputStream.java:39)
at org.jboss.resteasy.reactive.server.vertx.VertxResteasyReactiveRequestContext.createInputStream(VertxResteasyReactiveRequestContext.java:243)
at org.jboss.resteasy.reactive.server.core.ResteasyReactiveRequestContext.getInputStream(ResteasyReactiveRequestContext.java:717)
at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler.handle(FormBodyHandler.java:91)
... 9 moreThese are the libs I'm putting in the providers/ dir (via the shade plugin above):
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.15.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.15.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2 in the shaded jar.
[INFO] Including com.datastax.oss:java-driver-core:jar:4.15.0 in the shaded jar.
[INFO] Including com.datastax.oss:native-protocol:jar:1.5.1 in the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.77.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.77.Final in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.77.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.77.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.77.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.77.Final in the shaded jar.
[INFO] Including com.datastax.oss:java-driver-shaded-guava:jar:25.1-jre-graal-sub-1 in the shaded jar.
[INFO] Including com.typesafe:config:jar:1.4.1 in the shaded jar.
[INFO] Including com.github.jnr:jnr-posix:jar:3.1.15 in the shaded jar.
[INFO] Including com.github.jnr:jnr-ffi:jar:2.2.11 in the shaded jar.
[INFO] Including com.github.jnr:jffi:jar:1.3.9 in the shaded jar.
[INFO] Including com.github.jnr:jffi:jar:native:1.3.9 in the shaded jar.
[INFO] Including org.ow2.asm:asm:jar:9.2 in the shaded jar.
[INFO] Including org.ow2.asm:asm-commons:jar:9.2 in the shaded jar.
[INFO] Including org.ow2.asm:asm-analysis:jar:9.2 in the shaded jar.
[INFO] Including org.ow2.asm:asm-tree:jar:9.2 in the shaded jar.
[INFO] Including org.ow2.asm:asm-util:jar:9.2 in the shaded jar.
[INFO] Including com.github.jnr:jnr-a64asm:jar:1.0.0 in the shaded jar.
[INFO] Including com.github.jnr:jnr-x86asm:jar:1.0.2 in the shaded jar.
[INFO] Including com.github.jnr:jnr-constants:jar:0.10.3 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.26 in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:4.1.18 in the shaded jar.
[INFO] Including org.hdrhistogram:HdrHistogram:jar:2.1.12 in the shaded jar.
[INFO] Including org.reactivestreams:reactive-streams:jar:1.0.3 in the shaded jar.
[INFO] Including com.github.stephenc.jcip:jcip-annotations:jar:1.0-1 in the shaded jar.
[INFO] Including com.github.spotbugs:spotbugs-annotations:jar:3.1.12 in the shaded jar.
[INFO] Including com.datastax.oss:java-driver-mapper-runtime:jar:4.15.0 in the shaded jar.
[INFO] Including com.datastax.oss:java-driver-query-builder:jar:4.15.0 in the shaded jar.
[INFO] Including org.cognitor.cassandra:cassandra-migration:jar:2.6.0_v4 in the shaded jar.Update. It was in fact a conflict. Removing the following from the shaded jar resolves it:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>io.netty:netty-handler</exclude>
<exclude>io.netty:netty-common</exclude>
<exclude>io.netty:netty-resolver</exclude>
<exclude>io.netty:netty-buffer</exclude>
<exclude>io.netty:netty-transport</exclude>
<exclude>io.netty:netty-codec</exclude>
<exclude>org.ow2.asm:asm</exclude>
<exclude>org.ow2.asm:asm-commons</exclude>
<exclude>org.ow2.asm:asm-analysis</exclude>
<exclude>org.ow2.asm:asm-tree</exclude>
<exclude>org.ow2.asm:asm-util</exclude>
<exclude>org.reactivestreams:reactive-streams</exclude>
<exclude>com.google.code.findbugs</exclude>
<exclude>com.fasterxml.jackson.core:jackson-core</exclude>
<exclude>com.fasterxml.jackson.core:jackson-databind</exclude>
<exclude>com.fasterxml.jackson.core:jackson-annotations</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
</filters>
</configuration>
</plugin>I'll put together a PR when I have some more time.
opdt
commented
11 months ago @xgp Please try the newest release and the newly provided docker-compose.yaml
xgp
commented
11 months ago @opdt It doesn't work. I used the following commands:
git clone git@github.com:opdt/keycloak-cassandra-extension.git
cd keycloak-cassandra-extension/
mvn clean install -DskipTests
docker compose upThere are a bunch of errors because the providers you've specified in the docker-compose.yaml don't exist. It looks like you've opted to do the overriding by using the Keycloak provider IDs in your providers. E.g. using legacy in your DatastoreProviderFactory implementation.
private static final String PROVIDER_ID = "legacy"; // Override legacy provider to disable timers / event listeners and stuff...I'm not sure those collisions are a good idea, as then people who use this extension can't really pick and choose which providers they want to override. Is there an issue with just naming them all cassandra or each one cassandra-spi-it-overrides?
Anyway, I tried to figure out all of the changes to make in the docker-compose.yaml file, but I'm still getting some errors. E.g. When I change the docker-compose.yaml to
diff --git a/docker-compose.yaml b/docker-compose.yaml
index acc5cd2..46ff721 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -33,16 +33,16 @@ services:
environment:
- KC_SPI_CONNECTIONS_INFINISPAN_QUARKUS_ENABLED=false
- KC_SPI_CONNECTIONS_INFINISPAN_DEFAULT_ENABLED=false
- - KC_SPI_USER_SESSIONS_PROVIDER=cassandra
- - KC_SPI_LOGIN_FAILURE_PROVIDER=cassandra
- - KC_SPI_AUTHENTICATION_SESSIONS_PROVIDER=cassandra
- - KC_SPI_SINGLE_USE_OBJECT_PROVIDER=cassandra
- - KC_SPI_GLOBAL_LOCK_PROVIDER=none
- - KC_SPI_STICKY_SESSION_ENCODER_PROVIDER=disabled
- - KC_SPI_DEPLOYMENT_STATE_PROVIDER=map
- - KC_SPI_PUBLIC_KEY_STORAGE_PROVIDER=map
+ #- KC_SPI_USER_SESSIONS_PROVIDER=cassandra
+ #- KC_SPI_LOGIN_FAILURE_PROVIDER=cassandra
+ #- KC_SPI_AUTHENTICATION_SESSION_PROVIDER=cassandra
+ #- KC_SPI_SINGLE_USE_OBJECT_PROVIDER=cassandra
+ - KC_SPI_GLOBAL_LOCK_PROVIDER=dblock
+ #- KC_SPI_STICKY_SESSION_ENCODER_PROVIDER=disabled
+ #- KC_SPI_DEPLOYMENT_STATE_PROVIDER=map
+ #- KC_SPI_PUBLIC_KEY_STORAGE_PROVIDER=map
- KC_SPI_CONNECTIONS_JPA_LEGACY_ENABLED=false
- - KC_SPI_DATASTORE_PROVIDER=cassandra
+ #- KC_SPI_DATASTORE_PROVIDER=legacy
- KC_SPI_CASSANDRA_CONNECTION_DEFAULT_PORT=9042
- KC_SPI_CASSANDRA_CONNECTION_DEFAULT_CONTACT_POINTS=cassandra
- KC_SPI_CASSANDRA_CONNECTION_DEFAULT_USERNAME=cassandra
it still doesn't find dblock for the globalLock provider, which makes me think there may be other problems:
java.lang.RuntimeException: Failed to find provider dblock for globalLockLet me know if you tested this a different way, and I'm missing something obvious.
opdt
commented
11 months ago @xgp Sorry my bad, there were changes missing to docker-compose.yaml. I double checked locally, should work now.
xgp
commented
11 months ago @opdt Thanks. That works!
Any more information on why you chose to use the Keycloak standard provider IDs (e.g. infinispan, legacy, jpa, default, dblock, etc.) rather than using your own values and then selecting the providers using env vars or command line flags? I know it may not be your use case, but it makes it hard to use only one of the extensions.
opdt
commented
11 months ago It's not pretty but the best solution for now. We will try to improve this in Keycloak and then adjust the implementation of this extension accordingly. Might take a little while though.
xgp
commented
10 months ago Thank you for the link and that context. Seems like a bit of a mess on the Keycloak side currently. In any case, thank you for the help getting started on this.
When I try to start according to the instructions, I get an exception of
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the queryin theDefaultCassandraConnectionProviderFactory.createKeyspaceIfNotExistsmethod.I'm running in docker with a single cassandra node. Anything I'm obviously doing wrong?
My keycloak configuration:
My cassandra configuration:
Full stacktrace: