camunda-community-hub / zeeqs

GraphQL API for Zeebe data
Apache License 2.0
62 stars 15 forks source link

Java heap space #504

Open HMubaireek opened 4 months ago

HMubaireek commented 4 months ago

Describe the bug I'm using zeeqs with hazelcast & postgresql database

Occasionally, I get this Java heap space error, and I have to delete the pod to get this working again.

To Reproduce Steps to reproduce the behavior:

  1. It's hard for me to find replication steps.

Expected behavior I expect this to be handled. Probably there's a memory leak somewhere?

GraphQL request N/A

Environment (please complete the following information):

Additional context

2024-05-28 12:56:56.826 ERROR 1 --- [nt_1.internal-3] c.h.s.i.executionservice.TaskScheduler   : hz.client_1 [dev] [5.3.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@321a7a56[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.w.c.request.async.WebAsyncManager    : Started async request
2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.w.c.request.async.WebAsyncManager    : Async error, dispatch to /graphql
2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : Exiting but response remains open for further handling
2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : "ASYNC" dispatch for POST "/graphql", parameters={}
2024-05-28 12:56:56.827 DEBUG 1 --- [nio-9000-exec-1] o.s.w.s.f.s.HandlerFunctionAdapter       : Resume with async result [java.lang.OutOfMemoryError: Java heap space]
2024-05-28 12:56:56.827 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : Unresolved failure from "ASYNC" dispatch: javax.servlet.ServletException: Async processing failed
2024-05-28 12:56:56.827 ERROR 1 --- [nio-9000-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.827 ERROR 1 --- [nio-9000-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Async processing failed] with root cause

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.827 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for POST "/error", parameters={}
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-1] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [*/*] and supported [application/json, application/*+json, application/json, application/*+json]
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-2] o.s.w.s.f.s.HandlerFunctionAdapter       : Resume with async result [java.lang.OutOfMemoryError: Java heap space]
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : Unresolved failure from "ASYNC" dispatch: javax.servlet.ServletException: Async processing failed
2024-05-28 12:56:56.828 ERROR 1 --- [nio-9000-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.829 ERROR 1 --- [nio-9000-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Async processing failed] with root cause

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.829 DEBUG 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for POST "/error", parameters={}
2024-05-28 12:56:56.829 DEBUG 1 --- [nio-9000-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)
2024-05-28 12:56:56.829 DEBUG 1 --- [nio-9000-exec-2] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [*/*] and supported [application/json, application/*+json, application/json, application/*+json]
2024-05-28 12:56:56.830 DEBUG 1 --- [nio-9000-exec-2] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [{timestamp=Tue May 28 12:56:56 UTC 2024, status=500, error=Internal Server Error, path=/graphql}]
2024-05-28 12:56:58.517 DEBUG 1 --- [     parallel-1] o.s.w.s.adapter.NativeWebSocketSession   : Closing StandardWebSocketSession[id=8ad6e8e3-b148-cf73-ec94-55861214bf71, uri=wss://zeebe-zeeq.staging.work-permit.cntxt.ai/graphql]
2024-05-28 12:56:58.518 DEBUG 1 --- [     parallel-1] s.w.s.h.LoggingWebSocketHandlerDecorator : StandardWebSocketSession[id=8ad6e8e3-b148-cf73-ec94-55861214bf71, uri=wss://zeebe-zeeq.staging.work-permit.cntxt.ai/graphql] closed with CloseStatus[code=4408, reason=Connection initialisation timeout]
2024-05-28 12:57:01.941 DEBUG 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 500
2024-05-28 12:56:56.830 DEBUG 1 --- [nio-9000-exec-1] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [{timestamp=Tue May 28 12:56:56 UTC 2024, status=500, error=Internal Server Error, path=/graphql}]
2024-05-28 12:57:01.942 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 500
Exception in thread "Catalina-utility-2" java.lang.OutOfMemoryError: Java heap space
2024-05-28 12:57:58.225  WARN 1 --- [nt_1.internal-1] c.h.c.i.connection.tcp.HeartbeatManager  : hz.client_1 [dev] [5.3.6] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
2024-05-28 12:57:58.226  WARN 1 --- [nt_1.internal-1] c.h.c.i.c.tcp.TcpClientConnection        : hz.client_1 [dev] [5.3.6] ClientConnection{alive=false, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:57:58.225} closed. Reason: Heartbeat timed out

com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]

2024-05-28 12:58:04.593 DEBUG 1 --- [nio-9000-exec-5] org.hibernate.SQL                        :
    select
        processins0_.key_ as col_0_0_
    from
        process_instance processins0_
    where
        processins0_.process_definition_key=?
        and (
            processins0_.state in (
                ? , ? , ?
            )
        )
2024-05-28 12:58:06.707 ERROR 1 --- [pool-1-thread-1] i.z.h.connect.java.ZeebeHazelcast        : Fail to read from ring-buffer at sequence '78462'. Will try again.

com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=41, operation=Ringbuffer.ReadOne, isRetryable=true, correlationId=153252, messageType=170700, isEvent=false, isfragmented=false}, objectName = zeebe, target = partition 134, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-05-28 12:57:58.228. Start time: 2024-05-28 12:48:00.095. Total elapsed time: 598133 ms.
        at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:340) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:849) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:199) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]
        at ------ submitted from ------.() ~[na:na]
        at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:348) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapRuntimeException(AbstractInvocationFuture.java:1975) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapOrPeel(AbstractInvocationFuture.java:1956) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture$ExceptionalResult.wrapForJoinInternal(AbstractInvocationFuture.java:1438) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowForJoinInternal(AbstractInvocationFuture.java:601) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.joinInternal(AbstractInvocationFuture.java:585) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.proxy.ClientRingbufferProxy.invoke(ClientRingbufferProxy.java:220) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.proxy.ClientRingbufferProxy.readOne(ClientRingbufferProxy.java:156) ~[hazelcast-5.3.6.jar:5.3.6]
        at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.readNext(ZeebeHazelcast.java:119) ~[zeebe-hazelcast-connector-1.4.0.jar:1.4.0]
        at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.readFromBuffer(ZeebeHazelcast.java:111) ~[zeebe-hazelcast-connector-1.4.0.jar:1.4.0]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out
        at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:849) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:199) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]
Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        ... 10 common frames omitted

2024-05-28 12:58:06.708  INFO 1 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Removed connection to endpoint: [10.xx.x.x]:5701:dd88754e-d76d-493f-8ad0-6e27f83b491e, connection: ClientConnection{alive=false, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:57:58.225}
2024-05-28 12:58:06.709  WARN 1 --- [nt_1.internal-1] c.h.c.i.connection.tcp.HeartbeatManager  : hz.client_1 [dev] [5.3.6] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
2024-05-28 12:58:06.710  WARN 1 --- [nt_1.internal-1] c.h.c.i.c.tcp.TcpClientConnection        : hz.client_1 [dev] [5.3.6] ClientConnection{alive=false, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.710} closed. Reason: Heartbeat timed out

com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]

2024-05-28 12:58:06.711  INFO 1 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Removed connection to endpoint: [10.xx.5.5]:5701:e0923507-8901-4736-9a2b-84623e8dc861, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.710}
2024-05-28 12:58:06.712  WARN 1 --- [nt_1.internal-1] c.h.c.i.connection.tcp.HeartbeatManager  : hz.client_1 [dev] [5.3.6] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
2024-05-28 12:58:06.712  WARN 1 --- [nt_1.internal-1] c.h.c.i.c.tcp.TcpClientConnection        : hz.client_1 [dev] [5.3.6] ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.712} closed. Reason: Heartbeat timed out

com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]

2024-05-28 12:58:06.713  INFO 1 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Removed connection to endpoint: [10.xx.1.17]:5701:c3154107-c8e8-4b1a-956b-3aa6771e6adf, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.712}
2024-05-28 12:58:06.713  INFO 1 --- [nt_1.internal-1] com.hazelcast.core.LifecycleService      : hz.client_1 [dev] [5.3.6] HazelcastClient 5.3.6 (20231109 - 9903dc9) is CLIENT_DISCONNECTED
2024-05-28 12:58:23.537  INFO 1 --- [nt_1.internal-3] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Trying to connect to cluster: dev
2024-05-28 12:58:23.539  INFO 1 --- [nt_1.internal-3] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Trying to connect to Member [10.xx.5.5]:5701 - e0923507-8901-4736-9a2b-84623e8dc861
2024-05-28 12:58:31.946 ERROR 1 --- [nt_1.internal-3] c.h.s.i.executionservice.TaskScheduler   : hz.client_1 [dev] [5.3.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3125dee2[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:58:31.947 ERROR 1 --- [nt_1.internal-2] c.h.s.i.executionservice.TaskScheduler   : hz.client_1 [dev] [5.3.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5e10b6eb[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]

java.lang.OutOfMemoryError: Java heap space
saig0 commented 3 months ago

@HMubaireek thank you for reporting. :+1:

I didn't face an OutOfMemory error with ZeeQS before. To analyze the case, we must create a Java heap dump when the error occurs.

You should also check the resource limits and increase the memory if needed.