search

confluentinc / kafka-connect-storage-cloud

Kafka Connect suite of connectors for Cloud storage (Amazon S3)
Other
13 stars 329 forks source link

"java.lang.NoClassDefFoundError: com/google/common/base/Preconditions" when trying to run standalone mode #444

Closed dz902 closed 3 years ago

dz902 commented 3 years ago

Version confluentinc-kafka-connect-s3-10.0.1.

Here is the error:

[2021-08-08 02:25:15,588] ERROR WorkerSinkTask{id=s3-test-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover unt
il manually restarted. Error: com/google/common/base/Preconditions (org.apache.kafka.connect.runtime.WorkerSinkTask:607)
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
        at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:379)
        at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:392)
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:474)
        at org.apache.parquet.hadoop.ParquetWriter$Builder.<init>(ParquetWriter.java:345)
        at org.apache.parquet.avro.AvroParquetWriter$Builder.<init>(AvroParquetWriter.java:162)
        at org.apache.parquet.avro.AvroParquetWriter$Builder.<init>(AvroParquetWriter.java:153)
        at org.apache.parquet.avro.AvroParquetWriter.builder(AvroParquetWriter.java:43)
        at io.confluent.connect.s3.format.parquet.ParquetRecordWriterProvider$1.write(ParquetRecordWriterProvider.java:79)
        at io.confluent.connect.s3.format.KeyValueHeaderRecordWriterProvider$1.write(KeyValueHeaderRecordWriterProvider.java:105)
        at io.confluent.connect.s3.TopicPartitionWriter.writeRecord(TopicPartitionWriter.java:532)
        at io.confluent.connect.s3.TopicPartitionWriter.checkRotationOrAppend(TopicPartitionWriter.java:302)
        at io.confluent.connect.s3.TopicPartitionWriter.executeState(TopicPartitionWriter.java:245)                                                                   at io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:196)
        at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:234)                                                                                                at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:581)                                                                   at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:182)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:231)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

But all jars are there untouched:

accessors-smart-2.4.7.jar              httpclient-4.5.13.jar               kafka-connect-s3-10.0.1.jar
apacheds-i18n-2.0.0-M15.jar            httpcore-4.4.4.jar                  kafka-connect-storage-common-10.2.0.jar
apacheds-kerberos-codec-2.0.0-M15.jar  ion-java-1.0.2.jar                  kafka-connect-storage-common-htrace-core4-shaded-10.2.0.jar
api-asn1-api-1.0.0-M20.jar             jackson-annotations-2.10.5.jar      kafka-connect-storage-core-10.2.0.jar
api-util-1.0.0-M20.jar                 jackson-core-2.10.5.jar             kafka-connect-storage-format-10.2.0.jar
asm-3.1.jar                            jackson-core-asl-1.9.13.jar         kafka-connect-storage-partitioner-10.2.0.jar
asm-9.1.jar                            jackson-databind-2.10.5.1.jar       kafka-schema-registry-client-6.0.1.jar
audience-annotations-0.5.0.jar         jackson-dataformat-cbor-2.10.5.jar  kafka-schema-serializer-6.0.1.jar
avro-1.9.2.jar                         jackson-jaxrs-1.8.3.jar             netty-buffer-4.1.63.Final.jar
aws-java-sdk-core-1.11.1015.jar        jackson-xc-1.8.3.jar                netty-codec-4.1.63.Final.jar
aws-java-sdk-kms-1.11.1015.jar         jakarta.annotation-api-1.3.5.jar    netty-common-4.1.63.Final.jar
aws-java-sdk-s3-1.11.1015.jar          jakarta.inject-2.6.1.jar            netty-handler-4.1.63.Final.jar
aws-java-sdk-sts-1.11.1015.jar         jakarta.ws.rs-api-2.1.6.jar         netty-resolver-4.1.63.Final.jar
commons-beanutils-1.9.4.jar            javax.annotation-api-1.3.2.jar      netty-transport-4.1.63.Final.jar
commons-cli-1.4.jar                    java-xmlbuilder-0.4.jar             netty-transport-native-epoll-4.1.63.Final.jar
commons-codec-1.15.jar                 jaxb-api-2.3.0.jar                  netty-transport-native-unix-common-4.1.63.Final.jar
commons-collections-3.2.2.jar          jaxb-impl-2.2.3-1.jar               nimbus-jose-jwt-7.9.jar
commons-compress-1.19.jar              jcip-annotations-1.0-1.jar          osgi-resource-locator-1.0.3.jar
commons-configuration-1.6.jar          jersey-common-2.30.jar              parquet-avro-1.11.1.jar
commons-digester-1.8.jar               jersey-core-1.9.jar                 parquet-column-1.11.1.jar
commons-io-2.7.jar                     jersey-json-1.9.jar                 parquet-common-1.11.1.jar
commons-lang-2.6.jar                   jersey-server-1.9.jar               parquet-encoding-1.11.1.jar
commons-lang3-3.4.jar                  jets3t-0.9.0.jar                    parquet-format-structures-1.11.1.jar
commons-logging-1.2.jar                jettison-1.1.jar                    parquet-hadoop-1.11.1.jar
commons-math3-3.1.1.jar                jetty-6.1.26.jar                    protobuf-java-2.5.0.jar
commons-net-3.1.jar                    jetty-sslengine-6.1.26.jar          servlet-api-2.5.jar
commons-pool-1.6.jar                   jline-2.12.1.jar                    slf4j-api-1.7.30.jar
common-utils-6.0.1.jar                 jmespath-java-1.11.1015.jar         slf4j-log4j12-1.7.30.jar
curator-client-2.13.0.jar              joda-time-2.9.6.jar                 snappy-java-1.1.7.3.jar
curator-framework-2.13.0.jar           jsch-0.1.55.jar                     stax2-api-3.1.4.jar
curator-recipes-2.13.0.jar             json-smart-2.4.7.jar                swagger-annotations-1.6.2.jar
gson-2.8.6.jar                         jsp-api-2.1.jar                     woodstox-core-5.0.3.jar
hadoop-annotations-2.10.1.jar          jsr305-3.0.2.jar                    xmlenc-0.52.jar
hadoop-auth-2.10.1.jar                 kafka-avro-serializer-6.0.1.jar     zookeeper-3.5.8.jar
hadoop-common-2.10.1.jar               kafka-connect-avro-data-6.0.1.jar   zookeeper-jute-3.5.8.jar

plugin.path is correctly set, I could see the correct partition writer and file name:

[2021-08-08 02:25:15,564] INFO Opening record writer for: xxxxx/xxxxx.xxxxx.users/year=2021/month=08/day=07/xxxxx.xxxxx.tablename+0+0000000000.snappy.parquet
 (io.confluent.connect.s3.format.parquet.ParquetRecordWriterProvider:74)

Why is this happending? I think Guava is bundled with Hadoop jars and the jars are there. Please advise! Thanks.

dz902 commented 3 years ago

Config, which I don't think is much relevant to the problem

name=s3-test
rest.port=8084
connector.class=io.confluent.connect.s3.S3SinkConnector
storage.class=io.confluent.connect.s3.storage.S3Storage
s3.region=us-west-2
s3.bucket.name=xxxxx
topics.dir=xxxxx
topics.regex=xxxxx\..+
format.class=io.confluent.connect.s3.format.parquet.ParquetFormat
partitioner.class=io.confluent.connect.storage.partitioner.TimeBasedPartitioner
partition.duration.ms=600000
path.format='year'=YYYY/'month'=MM/'day'=dd
locale=en-US
timezone=Africa/Abidjan
flush.size=1

I read from Debezium.

dz902 commented 3 years ago

Arvo is working, only Parquet. This happens for 10.0.0 and 10.0.1, not sure why it seems no one has noticed.

dz902 commented 3 years ago

Problem solved. Add corresponding guava then enjoy.

https://stackoverflow.com/questions/68698325/confluent-kafka-s3-sink-connector-throws-java-lang-noclassdeffounderror-com-go/68698326#68698326