search

apache / rocketmq-flink

RocketMQ integration for Apache Flink. This module includes the RocketMQ source and sink that allows a flink job to either write messages into a topic or read from topics in a flink job.
https://rocketmq.apache.org/
Apache License 2.0
139 stars 88 forks source link

Some problems found by individuals during use #78

Open BukJiang opened 1 year ago

BukJiang commented 1 year ago

1. If you enable a new topic and a new tag, it seems that an exception will be thrown when the project starts

java.lang.RuntimeException: org.apache.rocketmq.client.exception.MQClientException: The message queue is not in assigned list, may be rebalancing, message queue: MessageQueue [topic=TEST_FLINK_TOPIC5, brokerName=DKF-RMQ-BK-1, queueId=2]

2. It seems that the offset of Rocketmq is not used correctly

Producer Log

16:29:24.738 [Thread-0] INFO com.buk.flink.rocketmq2.MqProducer - 【RocketMq 生产者】成功！content：Test Message 0，sendResult：SendResult [sendStatus=SEND_OK, msgId=7F0000014A9018B4AAC298E241760000, offsetMsgId=C0A81AC800002A9F0000000089E1D5FE, messageQueue=MessageQueue [topic=TEST_FLINK_TOPIC5, brokerName=DKF-RMQ-BK-1, queueId=0], queueOffset=21]

Consumer log Consumers will receive many identical messages

16:29:48.439 [rmq-pull-thread-0] DEBUG o.a.r.f.l.RocketMQSourceFunction - 7F0000014A9018B4AAC298E241760000_DKF-RMQ-BK-1 0 21 16:29:48.439 [rmq-pull-thread-0] INFO c.b.f.r.FlinkMqDataStreamConsumer - 收到消息：Test Message 0 16:29:48.439 [rmq-pull-thread-0] DEBUG o.a.r.f.l.RocketMQSourceFunction - 7F0000014A9018B4AAC298E241760000_DKF-RMQ-BK-1 0 21 16:29:48.439 [rmq-pull-thread-0] INFO c.b.f.r.FlinkMqDataStreamConsumer - 收到消息：Test Message 0 16:29:48.439 [rmq-pull-thread-0] DEBUG o.a.r.f.l.RocketMQSourceFunction - 7F0000014A9018B4AAC298E241760000_DKF-RMQ-BK-1 0 21 16:29:48.439 [rmq-pull-thread-0] INFO c.b.f.r.FlinkMqDataStreamConsumer - 收到消息：Test Message 0

After commenting out the following code, it will be normal image

deemogsw commented 1 year ago

please reset the branch of master to 90b00be02dd019ffed48cbc35cd050b07c4a36ab.The pr of #46 has some bugs and author has not fixed it yet. You can look the detail of this bug in issue https://github.com/apache/rocketmq-flink/issues/69#issuecomment-1295919302

wapmnw commented 1 year ago

Rocketmq 4.9.4 also has this issue when used, Is there a solution to the first problem?