search

openshift-knative / eventing-hyperfoil-benchmark

Apache License 2.0
2 stars 8 forks source link

Hyperfoil struggles to create a cluster with hundreds of receivers #44

Closed pierDipi closed 6 months ago

pierDipi commented 2 years ago

Steps to reproduce

pierDipi commented 2 years ago

Logs from the hyperfoil controller:

15:08:28,210 WARN  (TcpServer.Acceptor[7800]-2,hyperfoil-cluster-controller-8619) [o.j.p.TCP] JGRP000006: failed accepting connection from peer Socket[addr=/10.131.4.55,port=46321,localport=7800]: java.io.EOFException java.io.EOFException
    at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
    at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:247)
    at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:53)
    at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:129)
    at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:113)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:29,870 WARN  (jgroups-4,hyperfoil-cluster-controller-8619) [o.i.CLUSTER] ISPN000071: Caught exception when handling command TopologyJoinCommand{cacheName='org.infinispan.COUNTER', origin=p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncConsistentHashFactory@ffffd8e9, numSegments=256, numOwners=2, timeout=240000, cacheMode=DIST_SYNC, persistentUUID=98d694f3-5fd1-4a27-ae2a-0fb37c50923e, persistentStateChecksum=Optional.empty}, viewId=2} java.lang.NullPointerException
    at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
    at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
    at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
    at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:225)
    at org.infinispan.commands.topology.CacheJoinCommand.invokeAsync(CacheJoinCommand.java:42)
    at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:252)
    at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:174)
    at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:113)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1391)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1314)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:133)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1458)
    at org.jgroups.JChannel.up(JChannel.java:784)
    at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
    at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
    at org.jgroups.protocols.FlowControl.up(FlowControl.java:355)
    at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
    at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048)
    at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771)
    at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
    at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
    at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
    at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
    at org.jgroups.protocols.Discovery.up(Discovery.java:300)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1400)
    at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:32,556 WARN  (Connection.Receiver [10.131.4.56:7800 - 10.131.4.53:54443]-93,hyperfoil-cluster-controller-8619) [o.j.p.TCP] failed handling message java.io.IOException: Stream closed
    at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:275)
    at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:300)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:33,544 WARN  (Connection.Receiver [10.131.4.56:7800 - 10.130.2.40:32791]-106,hyperfoil-cluster-controller-8619) [o.j.p.TCP] failed handling message java.io.IOException: Stream closed
    at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:275)
    at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:300)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:33,561 INFO  (jgroups-18,hyperfoil-cluster-controller-8619) [o.j.threads] JBoss Threads version 2.3.3.Final
15:08:34,839 WARN  (Connection.Receiver [10.131.4.56:7800 - 10.129.4.42:52311]-120,hyperfoil-cluster-controller-8619) [o.j.p.TCP] failed handling message java.io.IOException: Stream closed
    at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:275)
    at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:300)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:34,962 WARN  (Connection.Receiver [10.131.4.56:7800 - 10.131.6.24:38015]-121,hyperfoil-cluster-controller-8619) [o.j.p.TCP] failed handling message java.io.IOException: Stream closed
    at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:275)
    at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:300)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:34,963 WARN  (Connection.Receiver [10.131.4.56:7800 - 10.131.2.41:37559]-118,hyperfoil-cluster-controller-8619) [o.j.p.TCP] failed handling message java.io.IOException: Stream closed
    at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:275)
    at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:300)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:35,213 INFO  (vert.x-eventloop-thread-0) [i.h.Hyperfoil] Deploying ControllerVerticle...
15:08:35,266 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Starting in directory /var/hyperfoil...
15:08:36,727 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerServer] Hyperfoil controller listening on https://hyperfoil-cluster-hyperfoil.apps.ci-ln-h8t3r12-72292.origin-ci-int-gce.dev.rhcloud.com
15:08:36,733 INFO  (vert.x-eventloop-thread-0) [i.h.Hyperfoil] ControllerVerticle deployed.
15:08:37,116 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-7-trigger-4-698ddf99b7-s8q94-50875, d41be873-b5bd-4a86-b92b-bf91234b45f8)
15:08:37,247 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-7-trigger-1-5cb896f999-87wvb-55154, 12ca2311-78f6-4240-a8e3-7dd17ea956df)
15:08:37,250 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215, ddbfe5f0-136d-47e6-83b9-fe1147995b18)
15:08:38,250 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, 3966a564-0601-4133-9010-aecd30f359d5)
15:08:38,253 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-6-trigger-4-f6f49894c-v7tk7-8308, 8dfa1477-4c59-4f04-81e7-18d484390d4d)
15:08:38,326 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-7-trigger-2-8447d54747-xs8h2-30523, 25c46c3b-b36e-481b-a22f-e7bf246c9081)
15:08:38,332 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, 3966a564-0601-4133-9010-aecd30f359d5)
15:08:39,049 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-6-trigger-5-774b98695-f5zcf-21829, f60cab0b-2f7b-4d51-a83a-d726b2d4055d)
15:08:39,134 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-7-trigger-3-8446b56979-m7xcg-20147, 92e07ca2-f317-46b0-a2c5-e41e92577dfb)
15:08:40,254 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-7-trigger-5-7d48b45bf5-7gds7-17251, baca39a9-1d5e-4ff3-80ea-dbc25157982f)
15:08:40,963 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-3-trigger-6-6fd9ccb459-4s9tw-25839, 728741cc-5d03-441b-81f2-60d6d2a4eddb)
15:08:41,270 INFO  (vert.x-eventloop-thread-1) [i.h.c.ControllerVerticle] Noticed auxiliary CE Receiver (node p10-r3-ord-7-trigger-6-6858f7499f-ltfmj-16459, 26e04c9c-17ec-46f7-ab14-339d10aa343c)
15:08:42,397 WARN  (jgroups-113,hyperfoil-cluster-controller-8619) [o.j.p.p.GMS] hyperfoil-cluster-controller-8619: failed to collect all ACKs (expected=21) for view [hyperfoil-cluster-controller-8619|19] after 2011 ms, missing 2 ACKs from (2) p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215
15:08:44,408 WARN  (jgroups-113,hyperfoil-cluster-controller-8619) [o.j.p.p.GMS] hyperfoil-cluster-controller-8619: failed to collect all ACKs (expected=22) for view [hyperfoil-cluster-controller-8619|20] after 2010 ms, missing 2 ACKs from (2) p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215
15:08:46,442 WARN  (jgroups-113,hyperfoil-cluster-controller-8619) [o.j.p.p.GMS] hyperfoil-cluster-controller-8619: failed to collect all ACKs (expected=25) for view [hyperfoil-cluster-controller-8619|21] after 2031 ms, missing 2 ACKs from (2) p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215
15:08:46,546 WARN  (vert.x-internal-blocking-1) [i.v.c.i.HAManager] Timed out waiting for group information to appear
15:08:46,748 WARN  (vert.x-internal-blocking-4) [i.v.c.i.HAManager] Timed out waiting for group information to appear
15:08:48,495 WARN  (jgroups-113,hyperfoil-cluster-controller-8619) [o.j.p.p.GMS] hyperfoil-cluster-controller-8619: failed to collect all ACKs (expected=29) for view [hyperfoil-cluster-controller-8619|22] after 2052 ms, missing 2 ACKs from (2) p10-r3-ord-1-trigger-4-76b664c84c-j4vgm-64965, p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215
15:08:49,867 WARN  (non-blocking-thread--p2-t2) [o.i.t.ClusterTopologyManagerImpl] ISPN000197: Error updating cluster member list for view 19, waiting for next view java.util.concurrent.CompletionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215 was suspected
    at org.infinispan.util.concurrent.CompletableFutures.asCompletionException(CompletableFutures.java:156)
    at org.infinispan.remoting.transport.impl.VoidResponseCollector.finish(VoidResponseCollector.java:67)
    at org.infinispan.remoting.transport.impl.VoidResponseCollector.finish(VoidResponseCollector.java:19)
    at org.infinispan.remoting.transport.impl.MultiTargetRequest.onNewView(MultiTargetRequest.java:140)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$receiveClusterView$3(JGroupsTransport.java:742)
    at org.infinispan.remoting.transport.impl.RequestRepository.lambda$forEach$0(RequestRepository.java:59)
    at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
    at org.infinispan.remoting.transport.impl.RequestRepository.forEach(RequestRepository.java:59)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$receiveClusterView$4(JGroupsTransport.java:742)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node p10-r3-ord-0-trigger-6-767b7b96bc-rvb2t-58215 was suspected
    at org.infinispan.remoting.transport.ResponseCollectors.remoteNodeSuspected(ResponseCollectors.java:31)
    at org.infinispan.remoting.transport.impl.VoidResponseCollector.addTargetNotFound(VoidResponseCollector.java:40)
    at org.infinispan.remoting.transport.impl.VoidResponseCollector.addTargetNotFound(VoidResponseCollector.java:19)
    at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:31)
    at org.infinispan.remoting.transport.impl.MultiTargetRequest.onNewView(MultiTargetRequest.java:130)
    ... 8 more

15:08:49,870 WARN  (non-blocking-thread--p2-t1) [o.i.CLUSTER] ISPN000071: Caught exception when handling command TopologyJoinCommand{cacheName='__vertx.haInfo', origin=p10-r3-ord-0-trigger-5-58c9d59ff-pz2dn-17160, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@3f620ea7, numSegments=256, numOwners=2, timeout=240000, cacheMode=REPL_SYNC, persistentUUID=f19c1f38-f341-4fae-bad1-3efcdcaaa762, persistentStateChecksum=Optional.empty}, viewId=23} java.lang.NullPointerException
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at org.infinispan.util.concurrent.ConditionFuture.checkConditions(ConditionFuture.java:139)
    at org.infinispan.util.concurrent.ConditionFuture.lambda$updateAsync$1(ConditionFuture.java:106)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:50,048 WARN  (non-blocking-thread--p2-t1) [o.i.CLUSTER] ISPN000071: Caught exception when handling command TopologyJoinCommand{cacheName='__vertx.haInfo', origin=p10-r3-ord-3-trigger-4-65b87f967-6fcsl-36731, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@17646988, numSegments=256, numOwners=2, timeout=240000, cacheMode=REPL_SYNC, persistentUUID=ce81f96f-6108-4700-a9c2-19372f8d327f, persistentStateChecksum=Optional.empty}, viewId=22} java.lang.NullPointerException
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at org.infinispan.util.concurrent.ConditionFuture.checkConditions(ConditionFuture.java:139)
    at org.infinispan.util.concurrent.ConditionFuture.lambda$updateAsync$1(ConditionFuture.java:106)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

15:08:50,049 WARN  (non-blocking-thread--p2-t1) [o.i.CLUSTER] ISPN000071: Caught exception when handling command TopologyJoinCommand{cacheName='__vertx.haInfo', origin=p10-r3-ord-8-trigger-1-755d5b6f6c-xz6bd-29330, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@1f3c5df6, numSegments=256, numOwners=2, timeout=240000, cacheMode=REPL_SYNC, persistentUUID=d487d866-fd88-4564-b579-e71d71cacb5e, persistentStateChecksum=Optional.empty}, viewId=21} java.lang.NullPointerException
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at org.infinispan.util.concurrent.ConditionFuture.checkConditions(ConditionFuture.java:139)
    at org.infinispan.util.concurrent.ConditionFuture.lambda$updateAsync$1(ConditionFuture.java:106)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
pierDipi commented 2 years ago

cc @willr3 @rvansa

pierDipi commented 2 years ago

Ok, this is the fatal exception in the hyperfoil controller

11:41:23,186 FATAL (Connection.Receiver [10.130.2.6:7800 - 10.128.4.13:58483]-1606,hyperfoil-cluster-controller-30793) [o.j.p.TCP] hyperfoil-cluster-controller-30793: thread pool is full (max=200, active=200); thread dump (dumped once, until thread_dump is reset):

full logs: hyperfoil-cluster-controller.txt

rvansa commented 2 years ago

This looks like a bug in Infinispan. Could you hack it to use later version (now it's 12.1.3.Final, there's a couple micros and also version 13 after that)?

pierDipi commented 2 years ago

Hi @rvansa, this is coming from the pod created by the Hyperfoil operator, are you suggesting bumping Hyperfoil to use a new version of infinispan or just our own custom receiver that is clustered with Hyperfoil?

It seems like 200 is their default maximum number of threads https://github.com/infinispan/infinispan/blob/86608f99f3ab64fb8fe7df74f499368eabbba553/core/src/main/resources/default-configs/default-jgroups-tcp.xml#L14, so that error might be expected, is there any way to tell Hyperfoil controller/operator to use a user-provided jgroups-tcp.xml config where we can bump this number and see how things go?

rvansa commented 2 years ago

You can never combine different versions of Infinispan (not even micros) in the same cluster, the protocol is not standardized. @pruivo traced NPE in the log to https://issues.redhat.com/browse/ISPN-13231which should be fixed in Infinispan 13.

JGroups config is taken from the classpath: https://github.com/Hyperfoil/Hyperfoil/blob/2a03cbdc1b84350a2a224c3f99e42d0886cf4e8d/clustering/src/main/resources/jgroups-tcp.xml#L13

So if you provided system-property jgroups.thread_pool.max_threads you could change that, but AFAIR the operator won't let you customize this. What you can do is to provide your own image by setting spec.image in the Hyperfoil resource.

pierDipi commented 2 years ago

You can never combine different versions of Infinispan (not even micros) in the same cluster, the protocol is not standardized. @pruivo traced NPE in the log to https://issues.redhat.com/browse/ISPN-13231which should be fixed in Infinispan 13.

Is it ok for you if I bump hyperfoil to use version 13?

JGroups config is taken from the classpath: https://github.com/Hyperfoil/Hyperfoil/blob/2a03cbdc1b84350a2a224c3f99e42d0886cf4e8d/clustering/src/main/resources/jgroups-tcp.xml#L13

So if you provided system-property jgroups.thread_pool.max_threads you could change that, but AFAIR the operator won't let you customize this. What you can do is to provide your own image by setting spec.image in the Hyperfoil resource.

Would you be open to exposing a way of customizing the controller pod created via the Hyperfoil CRD so that I can inject jgroups.thread_pool.max_threads system property?

I was also interested in specifying other PodSpec fields for that pod.

pierDipi commented 2 years ago

Is it ok for you if I bump hyperfoil to use version 13?

I tried 2 ways of getting infinispan version 13:

  1. exclude infinispan from vertx-infinispan
  2. bump vertx to 4.3.0

both options build successfully (I haven't run the tests), I'd prefer 2 but 2 has a bigger impact since vertx is used in multiple components in Hyperfoil, which one do you prefer so that I can open a PR in Hyperfoil?

pierDipi commented 2 years ago

option 2 in https://github.com/Hyperfoil/Hyperfoil/pull/224

rvansa commented 2 years ago

I also prefer to go with latest Vert.x. About the operator - sure, just don't expose only this particular property but let's make system properties configurable in general.