search

jgroups-extras / jgroups-aws

Discovery protocol for JGroups using AWS S3 buckets as cluster information store
http://jgroups.org
Apache License 2.0
13 stars 11 forks source link

Need help in configuring s3 ping for keycloak cache stack #210

Open Arulaln-AR opened 2 years ago

Arulaln-AR commented 2 years ago

Hi Team,

It is not an issue which i am raising here. While checking the keycloak distributed cache articles for aws ec2 instances, i landed up in the git page. I have gone through the read me document of this git page.

Still i am not sure, whether i have to use the "jgroups-aws/src/main/java/org/jgroups/protocols/aws/S3_PING.java" under my keycloak directory.

I follow the document like two things are clear to me.

1) Pom.xml changes 2) keycloak conf cache-ispn.xml chnages.

But certain other details are not clear to me, sorry if i am asking like a layman terms.

rhusar commented 2 years ago

Can you link the keycloak documentation you used?

I am assuming they haven't upgraded to JGroups 5.x, thus they using this version - https://github.com/jgroups-extras/jgroups-aws/tree/native-s3-ping-1.0.0.Final for which the configuration looks like this:

    <org.jgroups.aws.s3.NATIVE_S3_PING
            region_name="eu-west-1"
            bucket_name="jgroups-s3-test"
            bucket_prefix="jgroups"/>
Arulaln-AR commented 2 years ago

Hi Radoslav,

Main article from keycloak is "https://www.keycloak.org/server/caching". Under here, you can check the heading "Additional transport stacks"

Additional transport stacks

The following table shows transport stacks that are supported by Keycloak, but need some extra steps to work. Note that none of these stacks are Kubernetes / OpenShift stacks, so no need exists to enable the "google" stack if you want to run Keycloak on top of the Google Kubernetes engine. In that case, use the kubernetes stack. Instead, when you have a distributed cache setup running on AWS EC2 instances, you would need to set the stack to ec2, because ec2 does not support a default discovery mechanism such as UDP.

Stack name

Transport protocol

Discovery

ec2

TCP

NATIVE_S3_PING

google

TCP

GOOGLE_PING2

azure

TCP

AZURE_PING

Cloud vendor specific stacks have additional dependencies for Keycloak. For more information and links to repositories with these dependencies, see the Infinispan documentation https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport .

Followed the link " https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport" and ended up in the github page of yours.

On Tue, 26 Jul 2022 at 16:36, Radoslav Husar @.***> wrote:

Can you link the keycloak documentation you used?

I am assuming they haven't upgraded to JGroups 5.x, thus they using this version - https://github.com/jgroups-extras/jgroups-aws/tree/native-s3-ping-1.0.0.Final

— Reply to this email directly, view it on GitHub https://github.com/jgroups-extras/jgroups-aws/issues/210#issuecomment-1195340135, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP4YZ5MKF7MDS4HWP4LA733VV7BFFANCNFSM54VMCIJQ . You are receiving this because you authored the thread.Message ID: @.***>

-- Regards, Arulaln A R +91-8754438576

rhusar commented 2 years ago

I see, that documentation is lacking and is a bit sloppy. The https://www.keycloak.org/server/caching links to the 'dev' version of infinispan - https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport - as opposed to the one actually used, so the documentation doesn't match.

I assume what the documentation tells you is to do something like:

cd providers
wget https://repository.jboss.org/nexus/content/repositories/releases/org/jgroups/aws/s3/native-s3-ping/1.0.0.Final/native-s3-ping-1.0.0.Final.jar
bin/kc.sh build --cache-stack=ec2
Arulaln-AR commented 2 years ago

Hi Radoslav,

Thanks for the details. I am trying to use the same. Downloaded the jar file and then included the cache stack as ec2. Configured the cache-ispn.xml file. But there is something i am missing which is pom.xml, where do i change that?

If you are free, we can connect via zoom call and discuss this.

Please let me know.

On Tue, 26 Jul 2022 at 17:53, Radoslav Husar @.***> wrote:

I see, that documentation is lacking and is a bit sloppy. The https://www.keycloak.org/server/caching links to the 'dev' version of infinispan - https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport

  • as opposed to the one actually used, so the documentation doesn't match.

I assume what the documentation tells you is to do something like:

cd providers wget https://repository.jboss.org/nexus/content/repositories/releases/org/jgroups/aws/s3/native-s3-ping/1.0.0.Final/native-s3-ping-1.0.0.Final.jar bin/kc.sh https://repository.jboss.org/nexus/content/repositories/releases/org/jgroups/aws/s3/native-s3-ping/1.0.0.Final/native-s3-ping-1.0.0.Final.jarbin/kc.sh build --cache-stack=ec2

— Reply to this email directly, view it on GitHub https://github.com/jgroups-extras/jgroups-aws/issues/210#issuecomment-1195414097, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP4YZ5PY3OOXCVHHUSDMMZDVV7KERANCNFSM54VMCIJQ . You are receiving this because you authored the thread.Message ID: @.***>

-- Regards, Arulaln A R +91-8754438576

haroldpirum commented 2 years ago

Hi,

I had similar issue. To resolve I did the following:

These are customized, from default. should just be able to use stack ec2 option once keycloak infinispan version is updated.

COPY --chown=keycloak conf/cache-ispn-ec2.xml /opt/keycloak/conf/cache-ispn-ec2.xml COPY --chown=keycloak conf/custom-jgroups-ec2.xml /opt/keycloak/conf/custom-jgroups-ec2.xml RUN chmod 664 /opt/keycloak/conf/cache-ispn-ec2.xml RUN chmod 664 /opt/keycloak/conf/custom-jgroups-ec2.xml RUN /opt/keycloak/bin/kc.sh build --db=mysql --cache-config-file=cache-ispn-ec2.xml

- In custom cache-ispn-ec2.xml add jgroups section and define transport in cache-container:
<!-- custom stack goes into the jgroups element -->
<jgroups>
    <stack-file name="default-ec2" path="/opt/keycloak/conf/custom-jgroups-ec2.xml"/>
</jgroups>

<cache-container name="keycloak">
    <transport lock-timeout="60000" stack="default-ec2"/>
    <local-cache name="realms">

- define your custom-jgroups-ec2.xml as per the example in readme file.

Regards,
Harold
Msquared63 commented 1 year ago

Hi.

I've been struggle with this as well. I did everything above but now I get:

java.lang.NullPointerException: Cannot invoke "org.infinispan.commons.configuration.io.ConfigurationResourceResolver.resolveResource(String)" because "resourceResolver" is null

What am I missing? In keycloak, where does that pom.xml file go?

Thanks

Mike

serhiiKalchenko commented 1 year ago

Keycloak: 20.0.3 To implement default "ec2" Infinispan stacks protocol you should:

  1. Put these files in ./providers dir
    • jgroups-aws-2.0.1.Final.jar (stacks protocol)
    • aws-java-sdk-core-1.12.410.jar (access to AWS creds, etc.)
    • aws-java-sdk-s3-1.12.410.jar (access to S3, etc.)
    • joda-time-2.12.2.jar (is used)
  2. JAVA_OPTS_APPEND='-Djgroups.s3.region_name=us-east-1 -Djgroups.s3.bucket_name='
    • IAM profile role should be applied to EC2 instance (for AWS creds) and S3 bucket created
  3. Build Keycloak with option --cache-stack=ec2 (no --cache-config-file option!)
    • bin/kc.[sh|bat] build --cache-stack=ec2

By me it's working) GL & HF!

Logs:

2023-02-21 09:47:28,062 INFO  [org.infinispan.server.core.transport.EPollAvailable] (keycloak-cache-init) ISPN005028: Native Epoll transport not available, using NIO instead: java.lang.ExceptionInInitializerError
2023-02-21 09:47:28,453 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-02-21 09:47:28,472 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-02-21 09:47:28,506 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-02-21 09:47:28,987 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-02-21 09:47:29,086 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-02-21 09:47:29,270 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-02-21 09:47:29,290 WARN  [org.jgroups.stack.Configurator] (keycloak-cache-init) NATIVE_S3_PING has been deprecated; please upgrade to a newer version of the protocol
2023-02-21 09:47:30,181 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) using Amazon S3 ping in region us-east-1 with bucket 'my-jgroups-s3-bucket-test' and prefix ''
2023-02-21 09:47:30,937 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) found bucket my-jgroups-s3-bucket-test
2023-02-21 09:48:04,101 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [ip-10-68-49-170-40943|3] (2) [ip-10-68-49-170-40943, ip-10-68-49-190-31671]
2023-02-21 09:48:04,111 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `ip-10-68-49-190-31671`, physical addresses are `[10.68.49.190:7800]`

2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 78.156s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Profile prod activated.
mathieutu commented 1 year ago

@serhiiKalchenko I scoured the web and saw countless answers. Yours was the one which let me have Keycloak working with S3 discovery. Thank you so much. I'll probably edit this post soon with a full doc how to solve the full Keycloak distributed cache on Elastic Beanstalk EC2 Docker containers mystery.

shkmaaz11 commented 9 months ago

Keycloak: 20.0.3 To implement default "ec2" Infinispan stacks protocol you should:

  1. Put these files in ./providers dir
  • jgroups-aws-2.0.1.Final.jar (stacks protocol)
  • aws-java-sdk-core-1.12.410.jar (access to AWS creds, etc.)
  • aws-java-sdk-s3-1.12.410.jar (access to S3, etc.)
  • joda-time-2.12.2.jar (is used)
  1. JAVA_OPTS_APPEND='-Djgroups.s3.region_name=us-east-1 -Djgroups.s3.bucket_name='
  • IAM profile role should be applied to EC2 instance (for AWS creds) and S3 bucket created
  1. Build Keycloak with option --cache-stack=ec2 (no --cache-config-file option!)
  • bin/kc.[sh|bat] build --cache-stack=ec2

By me it's working) GL & HF!

Logs:

2023-02-21 09:47:28,062 INFO  [org.infinispan.server.core.transport.EPollAvailable] (keycloak-cache-init) ISPN005028: Native Epoll transport not available, using NIO instead: java.lang.ExceptionInInitializerError
2023-02-21 09:47:28,453 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-02-21 09:47:28,472 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-02-21 09:47:28,506 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-02-21 09:47:28,987 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-02-21 09:47:29,086 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-02-21 09:47:29,270 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-02-21 09:47:29,290 WARN  [org.jgroups.stack.Configurator] (keycloak-cache-init) NATIVE_S3_PING has been deprecated; please upgrade to a newer version of the protocol
2023-02-21 09:47:30,181 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) using Amazon S3 ping in region us-east-1 with bucket 'my-jgroups-s3-bucket-test' and prefix ''
2023-02-21 09:47:30,937 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) found bucket my-jgroups-s3-bucket-test
2023-02-21 09:48:04,101 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [ip-10-68-49-170-40943|3] (2) [ip-10-68-49-170-40943, ip-10-68-49-190-31671]
2023-02-21 09:48:04,111 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `ip-10-68-49-190-31671`, physical addresses are `[10.68.49.190:7800]`

2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 78.156s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Profile prod activated.

What is the health check you've configured in Target Group?

shkmaaz11 commented 9 months ago

@serhiiKalchenko I scoured the web and saw countless answers. Yours was the one which let me have Keycloak working with S3 discovery. Thank you so much. I'll probably edit this post soon with a full doc how to solve the full Keycloak distributed cache on Elastic Beanstalk EC2 Docker containers mystery.

Did you create a document on how to achieve it?

mathieutu commented 9 months ago

Indeed, thanks for the reminder!

I've written a blog article about putting Keycloak in production in our Beanstalk environment. Please let me know if it needs some more details! 🙂

pj-trimble commented 9 months ago

Keycloak: 20.0.3 To implement default "ec2" Infinispan stacks protocol you should:

  1. Put these files in ./providers dir
  • jgroups-aws-2.0.1.Final.jar (stacks protocol)
  • aws-java-sdk-core-1.12.410.jar (access to AWS creds, etc.)
  • aws-java-sdk-s3-1.12.410.jar (access to S3, etc.)
  • joda-time-2.12.2.jar (is used)
  1. JAVA_OPTS_APPEND='-Djgroups.s3.region_name=us-east-1 -Djgroups.s3.bucket_name='
  • IAM profile role should be applied to EC2 instance (for AWS creds) and S3 bucket created
  1. Build Keycloak with option --cache-stack=ec2 (no --cache-config-file option!)
  • bin/kc.[sh|bat] build --cache-stack=ec2

By me it's working) GL & HF!

Logs:

2023-02-21 09:47:28,062 INFO  [org.infinispan.server.core.transport.EPollAvailable] (keycloak-cache-init) ISPN005028: Native Epoll transport not available, using NIO instead: java.lang.ExceptionInInitializerError
2023-02-21 09:47:28,453 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-02-21 09:47:28,472 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-02-21 09:47:28,506 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-02-21 09:47:28,987 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-02-21 09:47:29,086 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-02-21 09:47:29,270 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-02-21 09:47:29,290 WARN  [org.jgroups.stack.Configurator] (keycloak-cache-init) NATIVE_S3_PING has been deprecated; please upgrade to a newer version of the protocol
2023-02-21 09:47:30,181 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) using Amazon S3 ping in region us-east-1 with bucket 'my-jgroups-s3-bucket-test' and prefix ''
2023-02-21 09:47:30,937 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) found bucket my-jgroups-s3-bucket-test
2023-02-21 09:48:04,101 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [ip-10-68-49-170-40943|3] (2) [ip-10-68-49-170-40943, ip-10-68-49-190-31671]
2023-02-21 09:48:04,111 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `ip-10-68-49-190-31671`, physical addresses are `[10.68.49.190:7800]`

2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 78.156s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Profile prod activated.

Thank you so much for this answer. This is the only configuration that works. Although, I was trying to implement this with KC23 and using the latest versions of the jars that are available on maven. And it kept failing with NoSuchBucketException. This is probably because of some breaking changes in the infinispan versions used between KC20 and 23. Has anyone got the latest version KC23 working with S3_PING?

shkmaaz11 commented 8 months ago

Keycloak: 20.0.3 To implement default "ec2" Infinispan stacks protocol you should:

  1. Put these files in ./providers dir
  • jgroups-aws-2.0.1.Final.jar (stacks protocol)
  • aws-java-sdk-core-1.12.410.jar (access to AWS creds, etc.)
  • aws-java-sdk-s3-1.12.410.jar (access to S3, etc.)
  • joda-time-2.12.2.jar (is used)
  1. JAVA_OPTS_APPEND='-Djgroups.s3.region_name=us-east-1 -Djgroups.s3.bucket_name='
  • IAM profile role should be applied to EC2 instance (for AWS creds) and S3 bucket created
  1. Build Keycloak with option --cache-stack=ec2 (no --cache-config-file option!)
  • bin/kc.[sh|bat] build --cache-stack=ec2

By me it's working) GL & HF! Logs:

2023-02-21 09:47:28,062 INFO  [org.infinispan.server.core.transport.EPollAvailable] (keycloak-cache-init) ISPN005028: Native Epoll transport not available, using NIO instead: java.lang.ExceptionInInitializerError
2023-02-21 09:47:28,453 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-02-21 09:47:28,472 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-02-21 09:47:28,506 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-02-21 09:47:28,987 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-02-21 09:47:29,086 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-02-21 09:47:29,270 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-02-21 09:47:29,290 WARN  [org.jgroups.stack.Configurator] (keycloak-cache-init) NATIVE_S3_PING has been deprecated; please upgrade to a newer version of the protocol
2023-02-21 09:47:30,181 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) using Amazon S3 ping in region us-east-1 with bucket 'my-jgroups-s3-bucket-test' and prefix ''
2023-02-21 09:47:30,937 INFO  [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) found bucket my-jgroups-s3-bucket-test
2023-02-21 09:48:04,101 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [ip-10-68-49-170-40943|3] (2) [ip-10-68-49-170-40943, ip-10-68-49-190-31671]
2023-02-21 09:48:04,111 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `ip-10-68-49-190-31671`, physical addresses are `[10.68.49.190:7800]`

2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 78.156s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2023-02-21 09:48:07,087 INFO  [io.quarkus] (main) Profile prod activated.

Thank you so much for this answer. This is the only configuration that works. Although, I was trying to implement this with KC23 and using the latest versions of the jars that are available on maven. And it kept failing with NoSuchBucketException. This is probably because of some breaking changes in the infinispan versions used between KC20 and 23. Has anyone got the latest version KC23 working with S3_PING?

Yes I've got KC23 working with S3_PING

pj-trimble commented 8 months ago

Hi @shkmaaz11 Thanks for the quick response. Would you mind sharing the dockerfile/providers used and the configuration/environment variables used.

I am trying to run it on EC2, so how did you make the private IP's discoverable and also I am assuming that the IAM Instance Profile Role linked to the EC2 instance should be enough for the keycloak (running in a container) to be able to access S3.

Thanks.

shkmaaz11 commented 8 months ago

Hi @shkmaaz11 Thanks for the quick response. Would you mind sharing the dockerfile/providers used and the configuration/environment variables used.

I am trying to run it on EC2, so how did you make the private IP's discoverable and also I am assuming that the IAM Instance Profile Role linked to the EC2 instance should be enough for the keycloak (running in a container) to be able to access S3.

Thanks.

Please go through this. I've documented it here https://medium.com/@maaz11/configuring-keycloak-with-s3-ping-protocol-on-aws-ecs-fargate-with-postgresql-rds-81aea8824dc6 I'm running on Fargate and not EC2