search

apache / rocketmq-flink

RocketMQ integration for Apache Flink. This module includes the RocketMQ source and sink that allows a flink job to either write messages into a topic or read from topics in a flink job.
https://rocketmq.apache.org/
Apache License 2.0
139 stars 88 forks source link

Version below 4.6.0 does not support DefaultLitePullConsumer #69

Open deemogsw opened 1 year ago

deemogsw commented 1 year ago

DefaultMQPullConsumer has been instead of by DefaultLitePullConsumer.After I rebase master into my develop branch,some mistakes have occurred.And my company uses version 4.3

image

So I run the old version and everything is normal.I'm thing about the compatibility.After reading the documents,I found the following explanation.

image

Since not all version support LitePull.The generality of the code is very poor, and users need to spend a lot of effort merging the code to use the full functionality. If there is no particularly big improvement, is it necessary to replace way of pull? There are still many features to be added,some bugs to be fixed. See #63 So I think that resetting the commit of #46 is a good choice.After all of functions are available,It's not too late to replace CommonPull by LitePull in a new branch or tag.

deemogsw commented 1 year ago

@SteNicholas @zhouxinyu Please pay attention to this problem，thanks

deemogsw commented 1 year ago

Maybe It has another problems.For testing ,i run the V4.8 in docker container.But it is still failed

image

So I read the documents again.In Assign mode,consumer must assign MessageQueues before seek offset.

image

But #46 use like this:

image

It use a subscribe mode.It alse use seek() function. I guess if the exception is caused by an api usage error.So i make a test like this.

image
gj-zhang commented 1 year ago

is there any progress ?

deemogsw commented 1 year ago

is there any progress ? The original contributors disappeared and have not fix this problem.Maintainer also rarely manage. You can visit my homepage for project of rocketMQ-flink-connector.This is the latest version we use in production.

zhouxinyu commented 1 year ago

Hello @deemogsw, sorry for the late reply. If you have some fixes about the current version, could you please merge it to the upstream through a PR?

deemogsw commented 1 year ago

Hello @deemogsw, sorry for the late reply. If you have some fixes about the current version, could you please merge it to the upstream through a PR?

Sorry, I don't know much about lite pull.I can fix known problems, but I can't solve hidden problems because we use a lower version with defaultMQPull.So I suggest supporting this feature in the new branch. I just removed the problematic PR and add some updates in our internal version.All of patches have been merged into this or waiting for merging.

kevinnerd commented 10 months ago

is there any progress ? The original contributors disappeared and have not fix this problem.Maintainer also rarely manage. You can visit my homepage for project of rocketMQ-flink-connector.This is the latest version we use in production.

Hello @deemogsw , your rocketMQ-flink-connector project solved my problem, thanks a lot~~ save me a lot of time.