Infinispan clustering is not working on EKS cluster

[jbharti]

I deploy the infinispan cluster using these following command:

helm lint ./infinispan-helm-charts helm install -n qa infinispan-server ./infinispan-helm-charts

And then port forward to access

kubectl port-forward service/infinispan-server 11222:11222 -n qa

On Minikube: Its working fine

When i run above command infinispan server got created and goes into cluster

on EKS : clustering is not happening When i run above command infinispan server got created but NOT goes into cluster

[ryanemerson]

@jbharti What's the status of the Infinispan pods?

[jbharti]

@ryanemerson : both pods are running, but created on two different fargate nodes

[ryanemerson]

Can you paste the yaml output of the Infinispan services and pods, as well as the logs of the Infinispan pods?

[jbharti]

================StatefulSet yaml===========

apiVersion: apps/v1 kind: StatefulSet metadata: annotations: meta.helm.sh/release-name: infinispan-server meta.helm.sh/release-namespace: qa openshift.io/display-name: Infinispan Cluster openshift.io/documentation-url: http://infinispan.org/documentation/ creationTimestamp: "2023-02-16T10:52:25Z" generation: 1 labels: app.kubernetes.io/managed-by: Helm app.kubernetes.io/version: 10.1.8 clusterName: infinispan-server helm.sh/chart: infinispan-0.3.0 meta.helm.sh/release-name: infinispan-server meta.helm.sh/release-namespace: qa name: infinispan-server namespace: qa resourceVersion: "696215873" uid: 96ed76c3-c750-4b26-9163-5bf5162706a7 spec: podManagementPolicy: OrderedReady replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app: infinispan-pod clusterName: infinispan-server serviceName: "" template: metadata: annotations: checksum/config: 1c19988646c7b1ffd3d1d1a9be3ab50ffd3a2d419e73c5e5c900feb209332542 checksum/identities: 6e19930c4ff897802b6192c21ff615e416293adebfac6f16663bc7a9fe279d57 creationTimestamp: null labels: app: infinispan-pod app.kubernetes.io/environment: qa clusterName: infinispan-server spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution:

• podAffinityTerm: labelSelector: matchLabels: app: infinispan-pod clusterName: infinispan-server topologyKey: kubernetes.io/hostname weight: 100 containers:
  • args:
    • --cluster-name=infinispan-server
    • --bind-address=0.0.0.0
    • -Djgroups.dns.query=infinispan-server-ping.qa.svc.cluster.local env:
    • name: JAVA_OPTIONS
    • name: IDENTITIES_BATCH value: /etc/security/identities-batch
    • name: SERVER_LIBS image: sk-infinispan-server:10.1.8 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 httpGet: path: rest/v2/cache-managers/default/health/status port: 11222 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 80 name: infinispan ports:
    • containerPort: 8888 name: ping protocol: TCP
    • containerPort: 11222 name: infinispan protocol: TCP
    • containerPort: 11223 name: infinispan-met protocol: TCP readinessProbe: failureThreshold: 5 httpGet: path: rest/v2/cache-managers/default/health/status port: 11222 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 80 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi startupProbe: failureThreshold: 60 httpGet: path: rest/v2/cache-managers/default/health/status port: 11222 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 80 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    • mountPath: /etc/config name: config-volume
    • mountPath: /opt/infinispan/server/data name: data-volume
    • mountPath: /etc/security name: identities-volume dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes:
  • configMap: defaultMode: 420 name: infinispan-server-configuration name: config-volume
  • name: identities-volume secret: defaultMode: 420 secretName: infinispan-server-generated-secret
  • emptyDir: {} name: data-volume updateStrategy: type: RollingUpdate status: availableReplicas: 2 collisionCount: 0 currentReplicas: 2 currentRevision: infinispan-server-79f5fdfb84 observedGeneration: 1 readyReplicas: 2 replicas: 2 updateRevision: infinispan-server-79f5fdfb84 updatedReplicas: 2

====================Service===================

apiVersion: v1 kind: Service metadata: annotations: meta.helm.sh/release-name: infinispan-server meta.helm.sh/release-namespace: qa creationTimestamp: "2023-02-16T10:52:24Z" labels: app: infinispan-service app.kubernetes.io/managed-by: Helm app.kubernetes.io/version: 10.1.8 clusterName: infinispan-server helm.sh/chart: infinispan-0.3.0 meta.helm.sh/release-name: infinispan-server meta.helm.sh/release-namespace: qa name: infinispan-server namespace: qa resourceVersion: "696209194" uid: df34a79a-5d45-4dad-be3b-1984098a7384 spec: clusterIP: 10.100.84.74 clusterIPs:

• 10.100.84.74 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies:
• IPv4 ipFamilyPolicy: SingleStack ports:
• name: infinispan nodePort: 30000 port: 11222 protocol: TCP targetPort: 11222 selector: app: infinispan-pod clusterName: infinispan-server sessionAffinity: None type: NodePort status: loadBalancer: {}

==========================log for infinispan-server-0==================== 10:53:54,922 INFO (main) [BOOT] JVM OpenJDK 64-Bit Server VM Eclipse Adoptium 11.0.18+10 10:53:54,928 INFO (main) [BOOT] JVM arguments = -Djgroups.tcp.address=176.24.5.93 -Djgroups.dns.query=infinispan-server-ping.qa.svc.cluster.local -Xms64m -Xmx512m -XX:MetaspaceSize=64M -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dinfinispan.server.home.path=/opt/infinispan-server 10:53:54,935 INFO (main) [BOOT] PID = 61 10:53:55,206 INFO (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting 10:53:55,206 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan-server/server/conf/infinispan.xml 10:53:55,206 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan-server/server/conf/log4j2.xml 10:53:58,226 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory' 10:53:58,226 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory' 10:53:58,234 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory' 10:54:02,322 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.8.Final 10:54:03,030 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel infinispan-server 10:54:05,742 INFO (main) [org.jgroups.protocols.pbcast.GMS] infinispan-server-0-37422: no members discovered after 2013 ms: creating cluster as coordinator 10:54:05,755 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel infinispan-server: [infinispan-server-0-37422|0] (1) [infinispan-server-0-37422] 10:54:05,785 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel infinispan-server local address is infinispan-server-0-37422, physical addresses are [176.24.5.93:7800] 10:54:05,933 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=10.1.8.Final timestamp=2023-02-16T10:54:05.928460Z 10:54:07,815 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager 10:54:16,311 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Protocol HotRod (internal) 10:54:16,808 INFO (main) [org.infinispan.SERVER] ISPN080018: Protocol REST (internal) 10:54:17,520 INFO (main) [org.infinispan.SERVER] ISPN080004: Protocol SINGLE_PORT listening on 0.0.0.0:11222 10:54:17,521 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'infinispan-server-0-37422' listening on http://0.0.0.0:11222 10:54:17,521 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 10.1.8.Final started in 22314ms

==========================log for infinispan-server-1====================

10:56:23,914 INFO (main) [BOOT] JVM OpenJDK 64-Bit Server VM Eclipse Adoptium 11.0.18+10 10:56:23,918 INFO (main) [BOOT] JVM arguments = -Djgroups.tcp.address=176.24.5.76 -Djgroups.dns.query=infinispan-server-ping.qa.svc.cluster.local -Xms64m -Xmx512m -XX:MetaspaceSize=64M -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dinfinispan.server.home.path=/opt/infinispan-server 10:56:23,994 INFO (main) [BOOT] PID = 61 10:56:24,201 INFO (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting 10:56:24,202 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan-server/server/conf/infinispan.xml 10:56:24,202 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan-server/server/conf/log4j2.xml 10:56:26,994 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory' 10:56:26,995 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory' 10:56:26,996 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory' 10:56:31,202 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.8.Final 10:56:31,794 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel infinispan-server 10:56:34,494 INFO (main) [org.jgroups.protocols.pbcast.GMS] infinispan-server-1-27275: no members discovered after 2078 ms: creating cluster as coordinator 10:56:34,518 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel infinispan-server: [infinispan-server-1-27275|0] (1) [infinispan-server-1-27275] 10:56:34,598 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel infinispan-server local address is infinispan-server-1-27275, physical addresses are [176.24.5.76:7800] 10:56:34,716 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=10.1.8.Final timestamp=2023-02-16T10:56:34.712548Z 10:56:36,394 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager 10:56:44,994 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Protocol HotRod (internal) 10:56:45,690 INFO (main) [org.infinispan.SERVER] ISPN080018: Protocol REST (internal) 10:56:46,402 INFO (main) [org.infinispan.SERVER] ISPN080004: Protocol SINGLE_PORT listening on 0.0.0.0:11222 10:56:46,402 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'infinispan-server-1-27275' listening on http://0.0.0.0:11222 10:56:46,403 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 10.1.8.Final started in 22200ms

---

we use custom image because we use our cache.xml

[ryanemerson]

@jbharti I don't have access to an EKS cluster, so it's not possible for me to try to reproduce.

Is it possible for you to try with a more recent Infinispan version? The 10.1.x stream is very old and is not supported upstream anymore. There have been many improvements and bug fixes to JGroups since this version, which may resolve the clustering issue.

[jbharti]

@ryanemerson : Yeah Its working fine for infinispan version 14.0.0. And clustering is happening in EKS too.

But It is our product need to use only infinispan 10.1.8. And it should be run in eks cluster.

So can we try to configuration Infinispan 10.1.8 with JGroup Jar of Infinispan 14.0.0 ?

or other configuration also need to change.

[tristantarrant]

Replacing components like that is not going to work. JGroups 5.x is a major upgrade that requires many changes in infinispan's transport code.