search

apache / camel-karavan

Apache Camel Karavan a Low-code Data Integration Platform
https://camel.apache.org
Apache License 2.0
417 stars 143 forks source link

Karavan can't connect to infinispan even though pods running fine #1027

Closed BassemAbdelFatah closed 8 months ago

BassemAbdelFatah commented 9 months ago

web designer show infinite loading in the infinispan as below image

and here is the karavan pod logs

2023-12-12 10:18:12,535 INFO [org.apa.cam.kar.ser.KaravanService] (executor-thread-1) Starting Karavan in Kubernetes 2023-12-12 10:18:12,703 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:18:12,703 INFO [org.apa.cam.kar.git.GiteaService] (executor-thread-1) Creating Gitea Repository 2023-12-12 10:18:12,705 INFO [org.apa.cam.kar.git.GiteaService] (executor-thread-1) Creating Gitea User Token 2023-12-12 10:18:12,983 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004074: Native Epoll transport not available, using NIO instead: io.netty.channel.epoll.Epoll 2023-12-12 10:18:12,984 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004108: Native IOUring transport not available, using NIO instead: io.netty.incubator.channel.uring.IOUring 2023-12-12 10:18:13,018 INFO [io.quarkus] (main) karavan 4.1.0 on JVM (powered by Quarkus 3.4.3) started in 5.388s. Listening on: http://0.0.0.0:8080 2023-12-12 10:18:13,019 INFO [io.quarkus] (main) Profile public activated. 2023-12-12 10:18:13,019 INFO [io.quarkus] (main) Installed features: [cdi, infinispan-client, jgit, jsch, kubernetes, kubernetes-client, oidc, openshift-client, quinoa, qute, resteasy-reactive, resteasy-reactive-jackson, scheduler, security, security-properties-file, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-openapi, smallrye-reactive-messaging, swagger-ui, vertx] 2023-12-12 10:18:14,492 INFO [org.apa.cam.kar.git.GiteaService] (executor-thread-1) Gitea User Token received 2023-12-12 10:18:14,676 INFO [org.apa.cam.kar.git.GiteaService] (executor-thread-1) Deleting Gitea User Token 2023-12-12 10:18:15,012 INFO [org.apa.cam.kar.git.GiteaService] (executor-thread-1) Deleted Gitea User Token 2023-12-12 10:18:15,013 INFO [org.apa.cam.kar.git.GiteaService] (executor-thread-1) Created Gitea Repository 2023-12-12 10:18:15,013 INFO [org.apa.cam.kar.kub.KubernetesService] (executor-thread-1) Stop Kubernetes Informers 2023-12-12 10:18:15,013 INFO [org.apa.cam.kar.kub.KubernetesService] (executor-thread-1) Starting Kubernetes Informers 2023-12-12 10:18:15,457 INFO [org.apa.cam.kar.kub.KubernetesService] (executor-thread-1) Started Kubernetes Informers 2023-12-12 10:18:18,383 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:18:20,208 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:18:25,229 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:18:27,289 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:18:32,307 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:18:34,468 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:18:39,491 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:18:41,621 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:18:46,635 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:18:48,645 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:18:53,659 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:18:55,819 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:00,835 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:02,933 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:07,951 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:10,018 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:15,039 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:16,867 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:21,895 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:24,015 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:29,031 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:30,924 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:35,943 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:37,802 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:42,819 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:44,757 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:49,797 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:51,987 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:19:57,007 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:19:58,946 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:03,969 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:05,896 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:10,912 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:12,884 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:17,903 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:19,927 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:24,943 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:27,002 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:32,016 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:34,173 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:39,195 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:41,137 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:46,155 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:47,958 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:20:52,972 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:20:55,097 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:21:00,118 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:21:01,978 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:21:06,998 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:21:09,043 INFO [org.apa.cam.kar.inf.InfinispanService] (executor-thread-2) InfinispanService is starting in remote mode 2023-12-12 10:21:14,055 INFO [org.inf.HOTROD] (executor-thread-2) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.17.Final 2023-12-12 10:21:14,058 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-2) Uncaught exception received by Vert.x: java.lang.RuntimeException: java.lang.Exception: Not connected... at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder.wrapIfNecessary(VertxEventBusConsumerRecorder.java:207) at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.handle(VertxEventBusConsumerRecorder.java:146) at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.handle(VertxEventBusConsumerRecorder.java:138) at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1512) at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29) at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.Exception: Not connected... at org.apache.camel.karavan.infinispan.InfinispanService.start(InfinispanService.java:119) at org.apache.camel.karavan.infinispan.InfinispanService.tryStart(InfinispanService.java:80) at org.apache.camel.karavan.infinispan.InfinispanService_Subclass.tryStart$$superforward(Unknown Source) at org.apache.camel.karavan.infinispan.InfinispanService_Subclass$$function$$1.apply(Unknown Source) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62) at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$3(FaultToleranceInterceptor.java:253) at io.smallrye.faulttolerance.core.InvocationContext.call(InvocationContext.java:20) at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29) at io.smallrye.faulttolerance.core.retry.Retry.doApply(Retry.java:88) at io.smallrye.faulttolerance.core.retry.Retry.apply(Retry.java:42) at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:255) at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:182) at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30) at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27) at org.apache.camel.karavan.infinispan.InfinispanService_Subclass.tryStart(Unknown Source) at org.apache.camel.karavan.service.KaravanService.startServices(KaravanService.java:134) at org.apache.camel.karavan.service.KaravanService_VertxInvoker_startServices_094be2f3e48c5e3b3465ddbb4f32b0fa4835c9c6.invokeBean(Unknown Source) at io.quarkus.vertx.runtime.EventConsumerInvoker.invoke(EventConsumerInvoker.java:45) at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.handle(VertxEventBusConsumerRecorder.java:142) ... 11 more

and infinispan logs 2023-12-12 09:57:55,359 INFO (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 17.0.8+7-LTS 2023-12-12 09:57:55,423 INFO (main) [BOOT] JVM arguments = [-server, --add-exports, java.naming/com.sun.jndi.ldap=ALL-UNNAMED, --add-opens, java.base/java.util=ALL-UNNAMED, --add-opens, java.base/java.util.concurrent=ALL-UNNAMED, -Xlog:gc*:file=/opt/infinispan/server/log/gc.log:time,uptimemillis:filecount=5,filesize=3M, -Djgroups.dns.query=infinispan-ping.default.svc.cluster.local, -Xmx512m, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.infinispan.server.loader.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-14.0.17.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap, --cluster-name=infinispan, --server-config=/etc/config/infinispan.yml, --logging-config=/etc/config/log4j2.xml, --bind-address=0.0.0.0] 2023-12-12 09:57:55,426 INFO (main) [BOOT] PID = 169 2023-12-12 09:57:55,523 INFO (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server 14.0.17.Final starting 2023-12-12 09:57:55,524 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /etc/config/infinispan.yml 2023-12-12 09:57:55,524 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /etc/config/log4j2.xml 2023-12-12 09:57:56,930 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory' 2023-12-12 09:57:56,930 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory' 2023-12-12 09:57:56,933 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory' 2023-12-12 09:57:56,934 WARN (main) [org.infinispan.SERVER] ISPN080059: No script engines are available 2023-12-12 09:57:57,962 INFO (main) [org.infinispan.SERVER] ISPN005054: Native IOUring transport not available, using NIO instead: java.lang.RuntimeException: failed to allocate memory for io_uring ring; try raising memlock limit (see getrlimit(RLIMIT_MEMLOCK, ...) or ulimit -l): Cannot allocate memory 2023-12-12 09:57:58,443 WARN (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal 2023-12-12 09:57:58,541 INFO (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller' 2023-12-12 09:58:00,351 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channelinfinispanwith stackkubernetes 2023-12-12 09:58:00,357 INFO (main) [org.jgroups.JChannel] local_addr: ac7c0013-5302-4e8c-b4fd-e76c640525e4, name: infinispan-0-21468 2023-12-12 09:58:00,425 INFO (main) [org.jgroups.protocols.FD_SOCK2] server listening on *.57800 2023-12-12 09:58:02,441 INFO (main) [org.jgroups.protocols.pbcast.GMS] infinispan-0-21468: no members discovered after 2004 ms: creating cluster as coordinator 2023-12-12 09:58:02,455 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel infinispan: [infinispan-0-21468|0] (1) [infinispan-0-21468] 2023-12-12 09:58:02,552 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channelinfinispanlocal address isinfinispan-0-21468, physical addresses are[10.244.1.64:7800] 2023-12-12 09:58:02,637 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=14.0.17.Final timestamp=2023-12-12T09:58:02.635741919Z 2023-12-12 09:58:03,245 INFO (main) [org.jboss.threads] JBoss Threads version 2.3.3.Final 2023-12-12 09:58:03,951 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager 2023-12-12 09:58:04,664 WARN (main) [org.infinispan.SERVER] ISPN080072: JMX remoting enabled without a default security realm. All connections will be rejected. 2023-12-12 09:58:04,736 INFO (main) [org.infinispan.server.core.telemetry.TelemetryServiceFactory] ISPN000953: OpenTelemetry integration is disabled 2023-12-12 09:58:04,952 INFO (ForkJoinPool.commonPool-worker-1) [org.infinispan.SERVER] ISPN080018: Started connector HotRod (internal) 2023-12-12 09:58:05,147 INFO (main) [org.infinispan.SERVER] ISPN080018: Started connector REST (internal) 2023-12-12 09:58:05,176 INFO (main) [org.infinispan.SERVER] Using transport: Epoll 2023-12-12 09:58:15,479 INFO (main) [org.infinispan.SERVER] ISPN080004: Connector SinglePort (default) listening on 0.0.0.0:11222 2023-12-12 09:58:15,479 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'infinispan-0-21468' listening on http://0.0.0.0:11222 2023-12-12 09:58:15,501 INFO (main) [org.infinispan.SERVER] ISPN080018: Started connector REST (internal) 2023-12-12 09:58:15,502 INFO (main) [org.infinispan.SERVER] Using transport: Epoll 2023-12-12 09:58:25,510 INFO (main) [org.infinispan.SERVER] ISPN080004: Connector SinglePort (metrics) listening on 0.0.0.0:11223 2023-12-12 09:58:25,511 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'infinispan-0-21468' listening on http://0.0.0.0:11223 2023-12-12 09:58:25,615 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 14.0.17.Final started in 30089ms

mgubaidullin commented 9 months ago

Infinispan pod does not look running fine:

Native IOUring transport not available, using NIO instead: java.lang.RuntimeException: failed to allocate memory for io_uring ring; try raising memlock limit (see getrlimit(RLIMIT_MEMLOCK, ...) or ulimit -l): Cannot allocate memory
BassemAbdelFatah commented 8 months ago

The issue was solved by rebooting the node that carries the pod.