In RocksDBConsumeQueueStore, there is a batching action when handling dispatchRequest. The original intention here is to improve efficiency, but in reality, it doesn't have much effect because RocksDB first writes to the memtable, which is a pure memory operation and doesn't incur much performance overhead. Additionally, this can lead to a bug where, in cases of fewer messages, batching might take a long time, resulting in message consumption delays. In severe cases, the last 15 messages might never be consumed.
For batching, our usual approach is to set a timeout, forcing a flush when the time exceeds the maximum timeout. However, putMessagePositionInfoWrapper does not have a dedicated thread for periodic flushing; instead, it is triggered by dispatchRequest. When there are no messages, setting a timeout is ineffective. Therefore, we should remove the batch directly in this case.
Before Creating the Bug Report
[X] I found a bug, not just asking a question, which should be created in GitHub Discussions.
[X] I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.
[X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
Runtime platform environment
all
RocketMQ version
all
JDK Version
all
Describe the Bug
RocksDBConsumeQueueStore 中在处理dispatchRequest时候会有一个攒批的动作,这里的初衷是为了提高效率,但是实际上并不会起到什么作用,因为rocksdb首先写入的memtable,这是纯内存操作,不会有多少性能损耗。而且这里还有导致一个bug,就是消息比较少的情况,攒批可能会比较久,这样会导致消息消费延迟,严重的话会导致最后15条消息一直消费不到。
In RocksDBConsumeQueueStore, there is a batching action when handling dispatchRequest. The original intention here is to improve efficiency, but in reality, it doesn't have much effect because RocksDB first writes to the memtable, which is a pure memory operation and doesn't incur much performance overhead. Additionally, this can lead to a bug where, in cases of fewer messages, batching might take a long time, resulting in message consumption delays. In severe cases, the last 15 messages might never be consumed.
Steps to Reproduce
对于攒批我们一般的处理方式是设置超时时间,当时间超过最大超市时间则强制刷新,但是putMessagePositionInfoWrapper没有专门的线程去定时刷新,而是由dispatchRequest触发的,当没有消息的时候,设置超时时间也没有用,所以我们在这里应该直接将这个batch去掉
For batching, our usual approach is to set a timeout, forcing a flush when the time exceeds the maximum timeout. However, putMessagePositionInfoWrapper does not have a dedicated thread for periodic flushing; instead, it is triggered by dispatchRequest. When there are no messages, setting a timeout is ineffective. Therefore, we should remove the batch directly in this case.
What Did You Expect to See?
nothing
What Did You See Instead?
nothing
Additional Context
nothing