CLAassistant
commented
3 months ago
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
mujingjing seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
现象: 在自建Apache Kafka 3.1.0的集群中，使用本connect从kafka同步数据到starrocks中，报错：(AbstractConfig.java:113)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:146)
at org.apache.kafka.connect.runtime.TaskConfig.(TaskConfig.java:51)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:607)
at org.apache.kafka.connect.runtime.Worker.startSinkTask(Worker.java:521)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startTask(StandaloneHerder.java:392)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.createConnectorTasks(StandaloneHerder.java:385)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.createConnectorTasks(StandaloneHerder.java:379)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.updateConnectorTasks(StandaloneHerder.java:436)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.lambda$putConnectorConfig$2(StandaloneHerder.java:231)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: com.starrocks.data.load.stream.StreamLoadDataFormat
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:136)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 21 more
java.lang.NoClassDefFoundError: com/starrocks/data/load/stream/StreamLoadDataFormat at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:735) at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483) at org.apache.kafka.common.config.AbstractConfig.
原因： 原因是org.apache.kafka.connect.runtime.isolation.PluginClassLoader实现问题，要求connector用到jar都必须在一个包里，否则无法加载class。配置的connector.class=com.starrocks.connector.kafka.StarRocksSinkConnector在starrocks-stream-load-sdk-1.0-20231130.060553-22-jar-with-dependencies.jar中，导致类无法加载找不到。
解决方法： 在打包时将starrocks-stream-load-sdk-1.0-20231130.060553-22-jar-with-dependencies.jar的代码shade进当前代码中，并在依赖中排除该jar包。
测试结果: 在自建Apache Kafka 3.1.0的集群中，测试通过。