search

wepay / kafka-connect-bigquery

DEPRECATED. PLEASE USE https://github.com/confluentinc/kafka-connect-bigquery. A Kafka Connect BigQuery sink connector
Apache License 2.0
155 stars 192 forks source link

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/cloud/bigquery/BigQuery #98

Open burd0047 opened 7 years ago

burd0047 commented 7 years ago

Hello,

I am trying to run kafka-connect-bigquery and getting the following error: [2017-10-27 20:07:07,792] INFO Loading plugin from: /usr/share/java/kafka-connect-bigquery/kcbq-connector-1.0.0-SNAPSHOT.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:176) Exception in thread "main" java.lang.NoClassDefFoundError: com/google/cloud/bigquery/BigQuery at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructor0(Class.java:3075) at java.lang.Class.newInstance(Class.java:412) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:242) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:223) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:190) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:150) at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47) at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:63) Caused by: java.lang.ClassNotFoundException: com.google.cloud.bigquery.BigQuery at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more

I am using the following DockerFile to build the image. I have also tried including the tar ball directly instead of building the code, but i get the same error.

FROM solsson/kafka-connect-jdbc@sha256:a6108f094eaef52469c8ca7d3e60b2523cadd6f5283f481c96308033b26fc92e WORKDIR /usr/src/app

RUN ["mkdir", "app"]

RUN ["mkdir", "/usr/logs"] RUN ["chmod", "a+rwx", "/usr/logs"]

RUN ["mkdir", "/usr/share/java/kafka-connect-bigquery"] RUN ["chmod", "a+rwx", "/usr/share/java/kafka-connect-bigquery"]

COPY . . RUN ["./gradlew", "clean", "confluentTarBall"] RUN ["tar", "-C", "/usr/share/java/kafka-connect-bigquery", "-xf", "/usr/src/app/bin/tar/kcbq-connector-1.0.0-SNAPSHOT-confluent-dist.tar"]

RUN ["rm", "-r", "app"]

WORKDIR /opt/kafka

Any ideas on what may be wrong? I tried googling for error, but didn't find much.

criccomini commented 6 years ago

Perhaps related to #105

machieltreffers commented 6 years ago

I'm running into the same error. Is there any news on this?

burd0047 commented 6 years ago

What do you have set for your plug in path?

jgao54 commented 6 years ago

I encountered the same error while running this locally. I suspect you are running a newer major version of confluent. Kafka-Connect-Bigquery requires confluent-3.0.0 specified in this repo, but I was running it with confluent-5.0.0 (which I think uses kafka 2.0.0 rather than 1.1.1), which resulted in NoClassDefFoundError: com/google/cloud/bigquery/BigQuery

If you must run confluent-5.0.0:

This is a bit risky as Kafka-Connect-Bigquery should be upgraded in lockstep with Debezium, which uses the older version of confluent as well. We haven't tested the pipeline in production with this newer version of confluent, so cannot be sure what kind of odd edge cases could come up when they mismatch.