CLAassistant
commented
3 months ago
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
mujingjing seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
现象： 使用目前的connect在自建Apache Kafka 3.1.0集群上面，同步数据到StarRocks。报错(AbstractConfig.java:113)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:146)
at org.apache.kafka.connect.runtime.TaskConfig.(TaskConfig.java:51)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:607)
at org.apache.kafka.connect.runtime.Worker.startSinkTask(Worker.java:521)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startTask(StandaloneHerder.java:392)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.createConnectorTasks(StandaloneHerder.java:385)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.createConnectorTasks(StandaloneHerder.java:379)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.updateConnectorTasks(StandaloneHerder.java:436)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.lambda$putConnectorConfig$2(StandaloneHerder.java:231)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: com.starrocks.data.load.stream.StreamLoadDataFormat
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:136)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 21 more
java.lang.NoClassDefFoundError: com/starrocks/data/load/stream/StreamLoadDataFormat at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:735) at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483) at org.apache.kafka.common.config.AbstractConfig.
原因： 经过调试发现是org.apache.kafka.connect.runtime.isolation.PluginClassLoader实现问题，这个classloader要求connector用到jar都必须在一个包里，否则无法加载class。由于com.starrocks.connector.kafka.StarRocksSinkConnector位于starrocks-stream-load-sdk-1.0-20231130.060553-22-jar-with-dependencies.jar中，所以导致了类找不到。
解决方法： 将starrocks-stream-load-sdk-1.0-20231130.060553-22-jar-with-dependencies.jar的代码在打包的时候shade进源代码中，在打包后的依赖中排查该jar。
测试： 修改打包方式后，在Apache Kafka 3.1.0中测试通过。