search

thingsboard / tbmq

Open-source, scalable, and fault-tolerant MQTT broker able to handle 4M+ concurrent client connections, supporting at least 3M messages per second throughput per single cluster node with low latency delivery. The cluster mode supports more than 100M concurrently connected clients.
https://thingsboard.io/products/mqtt-broker/
Apache License 2.0
570 stars 46 forks source link

[Bug] OutOfMemoryError - direct memory #105

Closed dmytro-landiak closed 5 months ago

dmytro-landiak commented 5 months ago

Describe the bug

2024-02-08T13:38:38.696891163Z 2024-02-08 13:38:38,696 [nioEventLoopGroup-3-11] ERROR o.t.m.b.server.MqttSessionHandler - [da3c567e-a2af-4d11-944c-b19ee420ef10] Unexpected Exception
2024-02-08T13:38:38.696905041Z java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 4190341064, limit: 4192206848)
2024-02-08T13:38:38.696908217Z  at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
2024-02-08T13:38:38.696920506Z  at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121)
2024-02-08T13:38:38.696923431Z  at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
2024-02-08T13:38:38.696925994Z  at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701)
2024-02-08T13:38:38.696928465Z  at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676)
2024-02-08T13:38:38.696931466Z  at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215)
2024-02-08T13:38:38.696933942Z  at io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:180)
2024-02-08T13:38:38.696936404Z  at io.netty.buffer.PoolArena.allocate(PoolArena.java:137)
2024-02-08T13:38:38.696938854Z  at io.netty.buffer.PoolArena.allocate(PoolArena.java:129)
2024-02-08T13:38:38.696941380Z  at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:400)
2024-02-08T13:38:38.696943935Z  at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
2024-02-08T13:38:38.696946432Z  at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
2024-02-08T13:38:38.696948891Z  at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
2024-02-08T13:38:38.696951536Z  at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
2024-02-08T13:38:38.696954253Z  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
2024-02-08T13:38:38.696956760Z  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
2024-02-08T13:38:38.696959276Z  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2024-02-08T13:38:38.696961878Z  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2024-02-08T13:38:38.696964595Z  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2024-02-08T13:38:38.696967098Z  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2024-02-08T13:38:38.696969587Z  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2024-02-08T13:38:38.696972994Z  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2024-02-08T13:38:38.696975796Z  at java.base/java.lang.Thread.run(Thread.java:833)

The above logs come up after some time of TBMQ running. It is not possible then to connect any client and send any message. The traffic is then at 0. The container is not crashing thus the broker is not restarting and is stuck in this unresponsive state.

Your Server Environment

Expected behavior MQTT communication remains stable.

Screenshots image image

Additional context Happens when there are clients that are publishing data to the unauthorized topics (i.e. "Client is not authorized to publish to the topic...")