apache / rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
https://rocketmq.apache.org/
Apache License 2.0
21.26k stars 11.7k forks source link

[Bug] Compensate for the missing file-based CQ during rollback. #8699

Open LetLetMe opened 2 months ago

LetLetMe commented 2 months ago

Before Creating the Bug Report

Runtime platform environment

all

RocketMQ version

all

JDK Version

all

Describe the Bug

在cq双写方案中,当双写切换到只读rocksdbcq时,当需要回滚的时候,offset和topic等可以通过json格式导出,下次broker启动时会自动marge加载(当然offset可能是时刻滚动的,这个过程中还是会出现少量的消息重复)。

同时,文件版cq的补偿构建也是需要做的,现在这块逻辑其实是有的,当broker启动改的时候会有recover逻辑,先recover cq,往前找三个文件,校验没问题后会更新最新消息物理位点,所有cq恢复完,会从所有cq中最大的消息物理位点开始,重放,重新dispatch;

n the CQ dual-write scheme, when switching to read-only RocksDB CQ, if a rollback is needed, the offset and topic information can be exported in JSON format. Upon the next broker startup, this information will be automatically merged and loaded (though the offset may change constantly, leading to a small amount of message duplication during this process).

Additionally, compensation construction for the file-based CQ is also necessary. Currently, this logic is already in place. When the broker starts, there is a recovery logic that first recovers the CQ, looks back at three files, and updates the latest message's physical position after verification. Once all CQs are recovered, replaying and redispatching will start from the largest message physical position among all CQs.

Steps to Reproduce

当回滚时补偿构建缺失的文件版cq

Compensate for the missing file-based CQ during rollback.

What Did You Expect to See?

nothing

What Did You See Instead?

nothing

Additional Context

nothing