search

apache / camel-kafka-connector

Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors
https://camel.apache.org
Apache License 2.0
154 stars 102 forks source link

NoClassDefFoundError when using S3 source connector with STS jar. #1536

Closed dezka closed 1 year ago

dezka commented 1 year ago

Version 3.18.2 of the connector, Kafka Connect 2.8, Java 11, Kafka 2.8.

config:

{
  "connector.class": "org.apache.camel.kafkaconnector.awss3source.CamelAwss3sourceSourceConnector",
  "camel.kamelet.aws-s3-source.deleteAfterRead": "true",
  "camel.kamelet.aws-s3-source.bucketNameOrArn": "mybucket",
  "topics": "mytopic",
  "tasks.max": "1",
  "name": "s3-source-test",
  "camel.kamelet.aws-s3-source.prefix": "test/",
  "camel.kamelet.aws-s3-source.useDefaultCredentialsProvider": "true",
  "camel.kamelet.aws-s3-source.region": "us-east-1"
}

I am intending to use the Web Identity Token method of authing into AWS. The token file is present on the Kafka Connect instances and I am currently using it with other connectors. However I seem to be missing another JAR dependency here and I can't figure out which one.

Files:

LICENSE.txt                                         camel-core-processor-3.18.2.jar                     http-client-spi-2.17.250.jar                        netty-codec-http-4.1.79.Final.jar
NOTICE.txt                                          camel-core-reifier-3.18.2.jar                       httpclient-4.5.13.jar                               netty-codec-http2-4.1.79.Final.jar
README.adoc                                         camel-direct-3.18.2.jar                             httpcore-4.4.15.jar                                 netty-common-4.1.79.Final.jar
annotations-13.0.jar                                camel-dsl-support-3.18.2.jar                        jackson-annotations-2.13.3.jar                      netty-handler-4.1.79.Final.jar
annotations-2.17.223.jar                            camel-health-3.18.2.jar                             jackson-core-2.13.3.jar                             netty-nio-client-2.17.250.jar
apache-client-2.17.250.jar                          camel-kafka-3.18.2.jar                              jackson-databind-2.13.3.jar                         netty-resolver-4.1.79.Final.jar
apicurio-registry-common-1.3.2.Final.jar            camel-kafka-connector-3.18.2.jar                    jackson-dataformat-avro-2.13.3.jar                  netty-transport-4.1.79.Final.jar
apicurio-registry-rest-client-1.3.2.Final.jar       camel-kamelet-3.18.2.jar                            jackson-datatype-jdk8-2.10.2.jar                    netty-transport-classes-epoll-4.1.79.Final.jar
apicurio-registry-utils-converter-1.3.2.Final.jar   camel-main-3.18.2.jar                               jakarta.activation-1.2.2.jar                        netty-transport-native-unix-common-4.1.79.Final.jar
apicurio-registry-utils-serde-1.3.2.Final.jar       camel-management-api-3.18.2.jar                     jakarta.activation-api-1.2.2.jar                    okhttp-3.14.9.jar
arns-2.17.250.jar                                   camel-seda-3.18.2.jar                               jakarta.xml.bind-api-2.3.3.jar                      okio-1.17.2.jar
auth-2.17.250.jar                                   camel-support-3.18.2.jar                            javax.annotation-api-1.3.2.jar                      profiles-2.17.250.jar
avro-1.11.0.jar                                     camel-util-3.18.2.jar                               jaxb-core-2.3.0.jar                                 protobuf-java-3.13.0.jar
aws-core-2.17.250.jar                               camel-xml-io-util-3.18.2.jar                        jaxb-impl-2.3.3.jar                                 protocol-core-2.17.250.jar
aws-query-protocol-2.17.250.jar                     camel-xml-jaxb-3.18.2.jar                           jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar            reactive-streams-1.0.3.jar
aws-xml-protocol-2.17.250.jar                       camel-xml-jaxp-3.18.2.jar                           jctools-core-3.3.0.jar                              regions-2.17.250.jar
camel-api-3.18.2.jar                                camel-yaml-dsl-3.18.2.jar                           json-utils-2.17.250.jar                             retrofit-2.9.0.jar
camel-aws-s3-source-kafka-connector-3.18.2.jar      camel-yaml-dsl-common-3.18.2.jar                    kotlin-reflect-1.3.20.jar                           s3-2.17.250.jar
camel-aws2-s3-3.18.2.jar                            camel-yaml-dsl-deserializers-3.18.2.jar             kotlin-stdlib-1.3.20.jar                            sdk-core-2.17.250.jar
camel-base-3.18.2.jar                               commons-codec-1.15.jar                              kotlin-stdlib-common-1.3.20.jar                     snakeyaml-engine-2.3.jar
camel-base-engine-3.18.2.jar                        commons-compress-1.21.jar                           medeia-validator-core-1.1.1.jar                     sts-2.17.250.jar
camel-bean-3.18.2.jar                               commons-logging-1.2.jar                             medeia-validator-jackson-1.1.1.jar                  third-party-jackson-core-2.17.250.jar
camel-core-engine-3.18.2.jar                        connect-json-2.6.0.jar                              metrics-spi-2.17.250.jar                            utils-2.17.223.jar
camel-core-languages-3.18.2.jar                     converter-jackson-2.9.0.jar                         netty-buffer-4.1.79.Final.jar
camel-core-model-3.18.2.jar                         eventstream-1.0.1.jar                               netty-codec-4.1.79.Final.jar

Here is the exception I'm getting:

java.lang.NoClassDefFoundError: software/amazon/awssdk/utils/cache/CachedSupplier$StaleValueBehavior
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:110)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:59)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider$BuilderImpl.build(InstanceProfileCredentialsProvider.java:377)
    at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.lambda$createChain$0(DefaultCredentialsProvider.java:104)
    at software.amazon.awssdk.utils.Lazy.getValue(Lazy.java:50)
    at software.amazon.awssdk.utils.Lazy.close(Lazy.java:71)
    at software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
    at software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
    at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.close(LazyAwsCredentialsProvider.java:50)
    at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.close(DefaultCredentialsProvider.java:128)
    at software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
    at software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
    at software.amazon.awssdk.utils.AttributeMap.lambda$close$0(AttributeMap.java:87)
    at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
    at software.amazon.awssdk.utils.AttributeMap.close(AttributeMap.java:87)
    at software.amazon.awssdk.core.client.config.SdkClientConfiguration.close(SdkClientConfiguration.java:79)
    at software.amazon.awssdk.core.internal.http.HttpClientDependencies.close(HttpClientDependencies.java:80)
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient.close(AmazonSyncHttpClient.java:73)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.close(BaseSyncClientHandler.java:88)
    at software.amazon.awssdk.services.s3.DefaultS3Client.close(DefaultS3Client.java:11047)
    at org.apache.camel.component.aws2.s3.AWS2S3Endpoint.doStop(AWS2S3Endpoint.java:141)
    at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
    at org.apache.camel.support.service.ServiceHelper.stopService(ServiceHelper.java:162)
    at org.apache.camel.support.service.ServiceHelper.stopAndShutdownServices(ServiceHelper.java:257)
    at org.apache.camel.support.service.ServiceHelper.stopAndShutdownServices(ServiceHelper.java:215)
    at org.apache.camel.impl.engine.RouteService.doShutdown(RouteService.java:302)
    at org.apache.camel.support.ChildServiceSupport.shutdown(ChildServiceSupport.java:113)
    at org.apache.camel.support.service.ServiceHelper.stopAndShutdownService(ServiceHelper.java:233)
    at org.apache.camel.impl.engine.AbstractCamelContext.shutdownServices(AbstractCamelContext.java:3629)
    at org.apache.camel.impl.engine.AbstractCamelContext.shutdownServices(AbstractCamelContext.java:3654)
    at org.apache.camel.impl.engine.AbstractCamelContext.doStop(AbstractCamelContext.java:3434)
    at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:152)
    at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
    at org.apache.camel.impl.engine.AbstractCamelContext.stop(AbstractCamelContext.java:2703)
    at org.apache.camel.main.SimpleMain.doStop(SimpleMain.java:58)
    at org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain.doStop(CamelKafkaConnectMain.java:67)
    at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
    at org.apache.camel.kafkaconnector.CamelSourceTask.stop(CamelSourceTask.java:303)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.close(WorkerSourceTask.java:173)
    at org.apache.kafka.connect.runtime.WorkerTask.doClose(WorkerTask.java:167)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:193)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)",
oscerd commented 1 year ago

What is the class missing? NoClassDefFoundError is the exception, what is the message?

dezka commented 1 year ago

Sorry, missed it on the OP, updated it to include the exception.

oscerd commented 1 year ago

The error is this one:

utils-2.17.223.jar

The StaleValueBehavior is not present in 2.17.223: https://javadoc.io/doc/software.amazon.awssdk/utils/2.17.223/software/amazon/awssdk/utils/cache/CachedSupplier.html

But it's present in 2.17.250: https://javadoc.io/doc/software.amazon.awssdk/utils/2.17.250/software/amazon/awssdk/utils/cache/CachedSupplier.html

oscerd commented 1 year ago

There should have been something wrong with 3.18.2, because utils is misaligned with the other jars. Try with 3.20.3, where they are all aligned to 2.18.40

oscerd commented 1 year ago

or alternatively you should remove the 2.17.233 version of utils jar and use 2.17.250

dezka commented 1 year ago

👍 thanks. Using 3.20.3 works for me. Is this a problem that only presented itself because I'm trying to use the STS jar? Or did 3.18.2 already have this issue?

Is 3.20.3 production ready?

oscerd commented 1 year ago

I think so. It should be related to sts.

This project is mostly followed by volunteers. It's just a release with a good level of trust

dezka commented 1 year ago

Cool. Thanks for all the help!