search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.9k stars 6.53k forks source link

skywalking oap 10.0.1 OutOfMemoryError: Java heap space #12531

Closed xinhuishuang closed 3 months ago

xinhuishuang commented 3 months ago

Search before asking

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

version：skywalking oap 10.0.1

After the oap service is running for some time, the service is abnormal, with both binary deployment and docker deployment.

2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.core.analysis.worker.MetricsAggregateWorker 145 [DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.6.Thread] ERROR [] - [10.0.1-6a9d727] Java heap space
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.storage.plugin.jdbc.common.dao.JDBCHistoryDeleteDAO 55 [pool-7-thread-1] INFO  [] - Deleting history data, ttl: 3, now: 1723599511829. Keep [20240812, 20240815]
2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.storage.plugin.jdbc.common.dao.JDBCHistoryDeleteDAO 55 [pool-7-thread-1] INFO  [] - Deleting history data, ttl: 3, now: 1723599511829. Keep [20240812, 20240815]
2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.storage.plugin.jdbc.common.dao.JDBCHistoryDeleteDAO 55 [pool-7-thread-1] INFO  [] - Deleting history data, ttl: 3, now: 1723599511829. Keep [20240812, 20240815]
2024-08-14 09:38:31,830 io.netty.channel.DefaultChannelPipeline 1152 [armeria-eventloop-epoll-5-29] WARN  [] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.ChannelPipelineException: com.linecorp.armeria.server.HttpServerPipelineConfigurator.handlerAdded() has thrown an exception; removed.
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:624) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.22.Thread" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.15.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.11.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.24.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.27.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.5.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.1.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.23.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.20.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HikariPool-1 housekeeper"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.2.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.18.Thread"
2024-08-14 09:40:36,979 io.netty.channel.DefaultChannelPipeline 1152 [armeria-eventloop-epoll-5-32] WARN  [] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.ChannelPipelineException: com.linecorp.armeria.server.HttpServerPipelineConfigurator.handlerAdded() has thrown an exception; removed.
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:624) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [netty-transport-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.108.Final.jar:4.1.108.Final]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-2-thread-1" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.28.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.31.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.7.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.16.Thread" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-2-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.22.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.6.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.8.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-4-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-10-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.0.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-9-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.31.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-executor-2433"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-2-thread-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-7-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.1.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.7.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.28.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "armeria-eventloop-epoll-5-21"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.29.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.16.Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "armeria-eventloop-epoll-5-29"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.13.Thread"
Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.29.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.13.Thread" Exception in thread "pool-5-thread-1" Exception in thread "armeria-eventloop-epoll-5-22" Exception in thread "armeria-eventloop-epoll-5-27" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:43:19,898 io.netty.channel.AbstractChannelHandlerContext 355 [armeria-boss-http-*:12800] WARN  [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:43:19,898 io.netty.channel.AbstractChannelHandlerContext 355 [armeria-eventloop-epoll-5-31] WARN  [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:43:19,898 io.netty.channel.AbstractChannelHandlerContext 355 [grpc-default-worker-ELG-10-1] WARN  [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:43:20,505 io.netty.channel.AbstractChannelHandlerContext 355 [armeria-eventloop-epoll-5-25] WARN  [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:43:20,505 io.netty.util.concurrent.AbstractEventExecutor 168 [armeria-eventloop-epoll-5-28] WARN  [] - A task raised an exception. Task: io.netty.channel.AbstractChannel$AbstractUnsafe$1@14f31166
java.lang.OutOfMemoryError: Java heap space
2024-08-14 09:43:21,164 io.netty.util.concurrent.AbstractEventExecutor 168 [armeria-eventloop-epoll-5-26] WARN  [] - A task raised an exception. Task: io.netty.channel.AbstractChannel$AbstractUnsafe$1@5289fd5c
java.lang.OutOfMemoryError: Java heap space
Aug 14, 2024 9:43:23 AM io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
        at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:523)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:921)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:629)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:472)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:510)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:255)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:166)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)

Aug 14, 2024 9:43:23 AM io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
io.netty.channel.StacklessClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space

Aug 14, 2024 9:43:23 AM io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
        at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:523)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:921)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:629)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:472)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:510)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:255)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:166)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)

Aug 14, 2024 9:43:25 AM io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
        at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432)
        at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:673)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)

Aug 14, 2024 9:43:25 AM io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
        at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481)
        at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432)
        at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:673)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-executor-2441"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-worker-ELG-10-2"
2024-08-14 12:14:34,091 io.netty.channel.AbstractChannelHandlerContext 355 [grpc-default-boss-ELG-9-1] WARN  [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space

What you expected to happen

Memory overflow

How to reproduce

After the oap service is running for some time, the service is abnormal, with both binary deployment and docker deployment. Try to configure the jvm parameter in the docker deployment mode, and the same exception occurred.

docker yaml docker run -d --name=skywalking-oap \ --restart=always \ -p 11800:11800 -p 12800:12800 \ -e SW_OAP_JAVA_OPTS='-Xms4g -Xmx4g' \ -e "TZ=Asia/Shanghai" \ apache/skywalking-oap-server:10.0.1

docker run -d --restart=always \ --name skywalking-ui \ --link skywalking-oap:skywalking-oap \ -p 8080:8080 \ -e SW_OAP_ADDRESS=http://skywalking-oap:12800 \ -e TZ=Asia/Shanghai \ apache/skywalking-ui:10.0.1

Anything else

No response

Are you willing to submit a pull request to fix on your own?

Code of Conduct

CodePrometheus commented 3 months ago

@xinhuishuang If you use default SW_STORAGE conf, the storage is H2 in memory mode, which will store all data in memory.

wu-sheng commented 3 months ago

You didn't read docs, I think? The default config leads all data in the memory rather than a persistent database.