【Bug】Source cannot consume new queues' messages when topic queue expanded

apache / rocketmq-flink

RocketMQ integration for Apache Flink. This module includes the RocketMQ source and sink that allows a flink job to either write messages into a topic or read from topics in a flink job.

https://rocketmq.apache.org/

Apache License 2.0

139 stars 88 forks source link

【Bug】Source cannot consume new queues' messages when topic queue expanded #95

Closed humkum closed 5 months ago

humkum commented 10 months ago

After topic expanded, Flink jobs cannot consume new queues。Because of follow reasons:

restoredOffsets doesn't have new queues' checkpoint
Flink job start with offset from restoredOffsets wouldn't initialize offset for new queues. And the consumer wouldn't pull messages from new queues because the offsetTable doesn't contain new queues.

lizhimins commented 7 months ago

Good catch