apache / shenyu

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance.
https://shenyu.apache.org/
Apache License 2.0
8.46k stars 2.94k forks source link

[BUG] <io.netty.util.internal.OutOfDirectMemoryError> #5800

Closed whirring closed 18 hours ago

whirring commented 1 day ago

Is there an existing issue for this?

Current Behavior

Same as Question#4274

I used Shenyu 2.4.1 in our production environment, for tps lower than 20 per second

this bug first appeared at 24.7.30, shenyu-bootstrap jvm options -Xmx2g -Xms2g -Xmn1g -Xss256k

then I temporarily changed jvm memory to -Xmx4g -Xms4g -Xmn1g -Xss256k

but at 24.11.21, this debug appeared again, logs goes like below

Expected Behavior

I think maybe ByteBuffer in netty not released finally

we only used divide, context_path, sign and monitor plugins

Steps To Reproduce

No response

Environment

ShenYu version(s): 2.4.1

Debug logs

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 4076863495, max: 4080271360) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:726) at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:681) at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) at io.netty.buffer.PoolArena.allocate(PoolArena.java:227) at io.netty.buffer.PoolArena.reallocate(PoolArena.java:397) at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:119) at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:310) at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:281) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1118) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1111) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1102) at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:96) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:394) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

Anything else?

No response

whirring commented 12 hours ago

is this bug similarity to this one reactor-netty-378

maybe something wrong in netty, or how can I do to fix this bug, just upgrade?