CLAassistant
commented
3 months ago
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
mujingjing seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
现象: 在自建Apache Kafka 3.1.0的集群中，使用本connect同步数据到starrocks，报错：(AbstractConfig.java:113)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:146)
at org.apache.kafka.connect.runtime.TaskConfig.(TaskConfig.java:51)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:607)
at org.apache.kafka.connect.runtime.Worker.startSinkTask(Worker.java:521)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startTask(StandaloneHerder.java:392)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.createConnectorTasks(StandaloneHerder.java:385)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.createConnectorTasks(StandaloneHerder.java:379)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.updateConnectorTasks(StandaloneHerder.java:436)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.lambda$putConnectorConfig$2(StandaloneHerder.java:231)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: com.starrocks.data.load.stream.StreamLoadDataFormat
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:136)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 21 mo
java.lang.NoClassDefFoundError: com/starrocks/data/load/stream/StreamLoadDataFormat at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:735) at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483) at org.apache.kafka.common.config.AbstractConfig.
错误原因: 原因是org.apache.kafka.connect.runtime.isolation.PluginClassLoader实现问题，要求connector用到jar都必须在一个包里，否则无法加载class。com.starrocks.connector.kafka.StarRocksSinkConnector在starrocks-stream-load-sdk-1.0-20231130.060553-22-jar-with-dependencies.jar中，无法加载
解决方案: 将starrocks-stream-load-sdk-1.0-20231130.060553-22-jar-with-dependencies.jar的代码shade中当前代码中。
测试结果: 在自建Apache Kafka 3.1.0的集群中,测试成功