search

GeyserMC / Geyser

A bridge/proxy allowing you to connect to Minecraft: Java Edition servers with Minecraft: Bedrock Edition.
https://geysermc.org
MIT License
4.7k stars 673 forks source link

LEAK: ByteBuf.release() was not called before it's garbage-collected #3853

Closed gre3x closed 2 months ago

gre3x commented 1 year ago

Describe the bug

My players are getting kicked when they join. I can see the player skin in the spigot server on my java account, but the bedrock client just gets a "Disconnected" message

It does not happen immediately. Only after a few hours or so and about 100 players, does this issue occur. This did not happen before 1.20 release. Restarting the Geyser Standalone fixes the problem temporarily

Full error from server:

[19:43:54.162 epollEventLoopGroup-5-6/INFO] Joe123 has disconnected from remote Java server on address ***.***.***.123 because of java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 133392804, limit: 134217728) [19:43:55.338 epollEventLoopGroup-3-4/ERROR] Error encoding packet LevelChunkPacket(chunkX=27, chunkZ=19, subChunksLength=17, cachingEnabled=false, requestSubChunks=false, subChunkLimit=0, blobIds=[]) java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 133392804, limit: 134217728) at java.nio.Bits.reserveMemory(Bits.java:178) ~[?:?] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[?:?] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[?:?] at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701) ~[Geyser.jar:?] at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676) ~[Geyser.jar:?] at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215) ~[Geyser.jar:?] at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197) ~[Geyser.jar:?] at io.netty.buffer.PoolArena.allocate(PoolArena.java:139) ~[Geyser.jar:?] at io.netty.buffer.PoolArena.reallocate(PoolArena.java:302) ~[Geyser.jar:?] at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:122) ~[Geyser.jar:?] at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305) ~[Geyser.jar:?] at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280) ~[Geyser.jar:?] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.codec.BaseBedrockCodecHelper.writeByteBuf(BaseBedrockCodecHelper.java:96) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.codec.v486.serializer.LevelChunkSerializer_v486.serialize(LevelChunkSerializer_v486.java:40) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.codec.v486.serializer.LevelChunkSerializer_v486.serialize(LevelChunkSerializer_v486.java:12) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.codec.BedrockCodec.tryEncode(BedrockCodec.java:78) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.netty.codec.packet.BedrockPacketCodec.encode(BedrockPacketCodec.java:39) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.netty.codec.packet.BedrockPacketCodec.encode(BedrockPacketCodec.java:19) ~[Geyser.jar:?] at io.netty.handler.codec.MessageToMessageCodec$1.encode(MessageToMessageCodec.java:67) ~[Geyser.jar:?] at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:90) ~[Geyser.jar:?] at io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116) ~[Geyser.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Geyser.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) ~[Geyser.jar:?] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) ~[Geyser.jar:?] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) ~[Geyser.jar:?] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:851) ~[Geyser.jar:?] at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1010) ~[Geyser.jar:?] at io.netty.channel.AbstractChannel.write(AbstractChannel.java:296) ~[Geyser.jar:?] at org.cloudburstmc.protocol.bedrock.BedrockPeer.onTick(BedrockPeer.java:87) ~[Geyser.jar:?] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [Geyser.jar:?] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:159) [Geyser.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [Geyser.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [Geyser.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [Geyser.jar:?] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406) [Geyser.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [Geyser.jar:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [Geyser.jar:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [Geyser.jar:?] at java.lang.Thread.run(Thread.java:831) [?:?]

To Reproduce

  1. Install latest Geyser Standalone
  2. Have around 100 players join over an hour or so
  3. Notice errors in logs

Expected behaviour

No exceptions and allow players to join

Screenshots / Videos

What players see if they try joining when it is having the issues:

image

Server Version and Plugins

Geyser Dump

https://dump.geysermc.org/jJI9eq3YGQnpf4D2hqiQ9HBuc8MJED2F

Geyser Version

2.1.1-SNAPSHOT (git-master-53d0026)

Minecraft: Bedrock Edition Device/Version

1.20.0

Additional Context

It does not happen immediately. Only after a few hours or so and about 100 players, does this issue occur. This did not happen before 1.20 release. Restarting the Geyser Standalone fixes the problem temporarily

gre3x commented 1 year ago

This has happened 5 times in the last day. The only thing that fixes it has been restarting the Geyser Standalone. It seems that there is somehow some memory/heap issue that is getting reached every couple of hours

gre3x commented 1 year ago

Now Bedrock players can join and chat, but they cant move. Switching to a different server under the proxy works, but does not solve the issue. Restarting was the only thing to fix it.

The exception above is getting spammed in the console logs

gre3x commented 1 year ago

it's not using a lot of RAM as far as I can tell. There is like 20 GB of unused memory. Heap issue?

onebeastchris commented 1 year ago

Could you try adding "-Dio.netty.leakDetection.level=PARANOID" to the java standalone startup arguments & see if you encounter any errors?

gre3x commented 1 year ago

I am working on that. In the mean time, I also tried switching to Geyser-Bungeecord and I got the same issue there @onebeastchris on a completetly different server. Here is the Heapdump from one of the times that the Bungee had the issue: https://drive.google.com/file/d/1dQCu4VGar8OcRODHB6YQysbC40wXB50j/view?usp=drive_link

Again, the only thing that fixed the Bungeecord was restarting it

gre3x commented 1 year ago

Seeing as this happened on both Geyser Standalone and Geyser BungeeCord (both separate servers/ips/etc), it seems like a memory leak? Is there anything I can do do at least make it so I don't have to keep restarting the server every few hours?

gre3x commented 1 year ago 

java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 133375567, limit: 134217728)
    at java.nio.Bits.reserveMemory(Bits.java:178) ~[?:?]
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[?:?]
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[?:?]
    at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701) ~[Geyser.jar:?]
    at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676) ~[Geyser.jar:?]
    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215) ~[Geyser.jar:?]
    at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197) ~[Geyser.jar:?]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:139) ~[Geyser.jar:?]
    at io.netty.buffer.PoolArena.reallocate(PoolArena.java:302) ~[Geyser.jar:?]
    at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:122) ~[Geyser.jar:?]
    at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305) ~[Geyser.jar:?]
    at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280) ~[Geyser.jar:?]
    at io.netty.buffer.WrappedByteBuf.ensureWritable(WrappedByteBuf.java:216) ~[Geyser.jar:?]
    at io.netty.buffer.AdvancedLeakAwareByteBuf.ensureWritable(AdvancedLeakAwareByteBuf.java:138) ~[Geyser.jar:?]
    at org.cloudburstmc.protocol.common.util.Zlib.inflate(Zlib.java:58) ~[Geyser.jar:?]
    at org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:35) ~[Geyser.jar:?]
    at org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:12) ~[Geyser.jar:?]
    at io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81) ~[Geyser.jar:?]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ~[Geyser.jar:?]
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[Geyser.jar:?]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[Geyser.jar:?]
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[Geyser.jar:?]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[Geyser.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[Geyser.jar:?]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[Geyser.jar:?]
    at org.cloudburstmc.netty.channel.raknet.RakChannelPipeline.lambda$onUnhandledInboundMessage$0(RakChannelPipeline.java:60) ~[Geyser.jar:?]
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [Geyser.jar:?]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [Geyser.jar:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [Geyser.jar:?]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403) [Geyser.jar:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [Geyser.jar:?]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [Geyser.jar:?]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [Geyser.jar:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[14:14:36.224 epollEventLoopGroup-5-13/ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    io.netty.buffer.AdvancedLeakAwareByteBuf.ensureWritable(AdvancedLeakAwareByteBuf.java:137)
    org.cloudburstmc.protocol.common.util.Zlib.inflate(Zlib.java:58)
    org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:35)
    org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:12)
    io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    org.cloudburstmc.netty.channel.raknet.RakChannelPipeline.lambda$onUnhandledInboundMessage$0(RakChannelPipeline.java:60)
    io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:833)
#2:
    io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:737)
    org.cloudburstmc.protocol.common.util.Zlib.inflate(Zlib.java:60)
    org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:35)
    org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:12)
    io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    org.cloudburstmc.netty.channel.raknet.RakChannelPipeline.lambda$onUnhandledInboundMessage$0(RakChannelPipeline.java:60)
    io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:833)
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
    io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:132)
    org.cloudburstmc.protocol.common.util.Zlib.inflate(Zlib.java:40)
    org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:35)
    org.cloudburstmc.protocol.bedrock.netty.codec.compression.ZlibCompressionCodec.decode(ZlibCompressionCodec.java:12)
    io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    org.cloudburstmc.netty.channel.raknet.RakChannelPipeline.lambda$onUnhandledInboundMessage$0(RakChannelPipeline.java:60)
    io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:833)
: 10 leak records were discarded because they were duplicates
: 501 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.```
gre3x commented 1 year ago

@onebeastchris the above leak errors spam my console after an hour or so with "-Dio.netty.leakDetection.level=PARANOID" added. The only way to stop them temporarily is to restart the server. Is there any way to fix this?

It seems to be related to something like "ByteBuf.release()" not being called (based on the error message)?

gre3x commented 1 year ago

@Chew I have responded. To replicate just install a normal Geyser Standalone or Bungee, add the flag, and have lots of people join it. It happens every few hours for me. Please remove waiting on response tag

gre3x commented 1 year ago

I have tried:

Nothing has solved the issue.

What do I do now? How do I resolve this? @onebeastchris @Chew

Konicai commented 1 year ago

Thank you for the information. You don't need to repeatedly mention people though - when anyone has more insight, it will be offered

Redned235 commented 1 year ago

Try updating to the latest build - I updated a few of our protocol dependencies which may resolve this issue.

gre3x commented 1 year ago

Sadly the issue is still occuring after upgrading to the latest version @Redned235. Is there anything else I can try to fix it?

gre3x commented 1 year ago

Is there anything else that I can provide to help resolve this?

I have been having to restart my network every few hours for nearly a week. I have tried everything that I can think of. It is getting very annoying for me, my staff, and my players to deal with this.

Redned235 commented 1 year ago

Unfortunately we do not have a resolution right now, but we’re actively looking into what caused this regression. Geyser logs may also help with hunting down this problem, which you’re welcome to share here or privately with us on Discord.

A heap dump won’t be all too useful in this case as the issue is the server running out of direct memory, rather than heap memory. You may be able to reduce the frequency by removing the MaxDirectMemorySize from your flags, since the JVM by default does not have a hard limit for this. It instead has a soft limit of 64MB and will allocate more in 32MB chunks. Keep in mind this is not a proper fix.

gre3x commented 1 year ago

Here are the logs from one of the recent times that this happened: https://mclo.gs/j1mX6GR

PedroMPagani commented 2 months ago

@gre3x has this been resolved?

gre3x commented 2 months ago

Yes - ultimately I just added another geyser standalone to lessen the load and it helped

PedroMPagani commented 2 months ago

It'd be nice to just close the issue then.

onebeastchris commented 2 months ago

As requested - ill close this issue then. If you encounter similar issues again, please provide as much information as possible and reach out to us here or on discord. Thanks!