search

apache / rocketmq-connect

A tool for scalable and reliably streaming data between Apache RocketMQ and other systems.
https://rocketmq.apache.org/
Apache License 2.0
121 stars 117 forks source link

cannot find class SourceConnector #130

Open adampan0527 opened 2 years ago

adampan0527 commented 2 years ago

Exception occurred while servicing http-request java.lang.NoClassDefFoundError: io/openmessaging/connector/api/source/SourceConnector

adampan0527 commented 2 years ago

Caused by: java.lang.ClassNotFoundException: io.openmessaging.connector.api.source.SourceConnector

adampan0527 commented 2 years ago

API调用： http://10.10.101.50:8081/connectors/replicatorTest?config={ "connector-class":"org.apache.rocketmq.replicator.RmqSourceReplicator", "source-rocketmq":"10.109.121.108:9876", "target-rocketmq":"10.109.11.140:9876", "replicator-store-topic":"replicatorTopic", "offset.sync.topic":"syncTopic", "taskDivideStrategy":"0", "white-list":"TopicTest,TopicTest2,fileTopic", "task-parallelism":"4", "connect-topicname":"replicatorTopic", "source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter" }

odbozhou commented 2 years ago

The latest replicator code has been merged, you can update the code and try again. If you have any questions, please communicate in time

adampan0527 commented 2 years ago

same problem occurred, topics are synced while messages are not. And it could not get name server, always found name server is null.

LittleBoy18 commented 2 years ago

https://github.com/apache/rocketmq-connect/pull/138 @odbozhou 发现个bug 修改后 topic可以同步 消息不能同步

zhangjidi2016 commented 2 years ago

this question is resolved in https://github.com/apache/rocketmq-connect/pull/134

odbozhou commented 2 years ago

Do you have any other questions?

LittleBoy18 commented 2 years ago

使用了最新的master代码，消息还是不能正常在两个mq集群之间同步. 发现消息其实被存储在source mq集群的 ”connect-topicname“ 这个topic里 看代码 并没有找到 targetMQAdminExt 执行 同步source mq 消息到 target mq的逻辑 RmqSourceReplicator 的能力和文档描述有出入， 是只同步topic元数据并不具有同步消息的能力？ @odbozhou @zhangjidi2016

LittleBoy18 commented 2 years ago

image 我的请求url, 按照文档操作创建 replicator-store-topic 这个参数的topic，发现并没有什么什么用处，消息被转存到了connect-topicname这个参数设定的topic

zhangjidi2016 commented 2 years ago

我测试是将待同步的topic放在“connect-topicname”这个参数中，可以同步消息。 image

LittleBoy18 commented 2 years ago

我测试的结果：是将source mq集群 白名单设置的topic中的消息 转存到了 souce mq集群 "connect-topicname"参数设置的topic里 我期待的结果是消息同步到了target mq集群里，您这边是可以做到这个效果的吗 @zhangjidi2016

zhangjidi2016 commented 2 years ago

target mq这边设置没用到，消息默认同步到connect.conf中指定的mq集群，我这边为了测试方便，源/目的/runtime使用的集群都是同一个集群 image

LittleBoy18 commented 2 years ago

我runtime和source mq是同一个集群，target mq是另外一套，消息没同步到target mq集群里

zhangjidi2016 commented 2 years ago

同步消息的producer在初始化时连接到的是runtime使用的mq集群，消息直接同步到这个集群中

LittleBoy18 commented 2 years ago

看代码我目前确实没找到同步到target mq集群的逻辑，看来是需要自己再实现一下了

odbozhou commented 2 years ago

我runtime和source mq是同一个集群，target mq是另外一套，消息没同步到target mq集群里

这样用法是有问题的，source是需要同步的集群A，runtime配置target mq集群B， 这样A集群的topic就可以同步到B集群。

LittleBoy18 commented 2 years ago

我runtime和source mq是同一个集群，target mq是另外一套，消息没同步到target mq集群里

这样用法是有问题的，source是需要同步的集群A，runtime配置target mq集群B， 这样A集群的topic就可以同步到B集群。

我按照你说的方法尝试了一下，消息是转存到了 runtime配置target mq集群B的connect-topicname里，消息并没有同步到 white-list 配置的topic中

LittleBoy18 commented 2 years ago

https://github.com/apache/rocketmq-connect/pull/145 readme 指导有错误，修改了点同步消息的逻辑，帮忙review一下吧 @odbozhou