search

apache / camel-kafka-connector-examples

Apache Camel Kafka Connector Examples
https://camel.apache.org
64 stars 43 forks source link

Camel AWS S3 source connector - Error org.reflections.ReflectionsException: Scanner SubTypesScanner was not configured #342

Closed farmountain closed 1 year ago

farmountain commented 1 year ago

While running the Kafka Connect - AWS S3 Source Connector

connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $KAFKA_HOME/config//opt/client/Kafka/kafka/config/camel-properties/docs/examples/CamelAwss3sourceSourceConnector.properties

[2022-11-16 13:37:19,764] INFO Loading plugin from: /opt/client/Kafka/kafka/config/camel-properties/camel-aws-s3-source-kafka-connector/kotlin-stdlib-common-1.3.20.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241) [2022-11-16 13:37:19,765] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:130) org.reflections.ReflectionsException: Scanner SubTypesScanner was not configured at org.reflections.Store.get(Store.java:39) at org.reflections.Store.get(Store.java:61) at org.reflections.Store.get(Store.java:46) at org.reflections.Store.getAll(Store.java:93) at org.reflections.Reflections.getSubTypesOf(Reflections.java:404) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:345) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:330) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:263) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:255) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:224) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:201) at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:79)

oscerd commented 1 year ago

What is the kafka version used?

farmountain commented 1 year ago

Kafka Version | 2.11-2.4.0

oscerd commented 1 year ago

and the camel-kafka-connector version?

farmountain commented 1 year ago

camel-aws-s3-source-kafka-connector-3..19.0-SNAPSHOT.jar

farmountain commented 1 year ago

same error when I try Camel Azure storage blob Source Connector

oscerd commented 1 year ago

We never seen something like this before. You need also to take into account this repository is not up-to-date with the latest release of ckc.

farmountain commented 1 year ago

ok, thanks for letting us know. May I know when can we expect the latest release? so that we can try again.

oscerd commented 1 year ago

There is no ETA for 3.19.0 release. First we need to release camel-kamelets 0.10.0 and then update on the ckc side.

oscerd commented 1 year ago

Can you please also show the configuration of the connector?

farmountain commented 1 year ago

name=CamelAws-s3-sourceSourceConnector connector.class=org.apache.camel.kafkaconnector.awss3source.CamelAwss3sourceSourceConnector tasks.max=1

use the kafka converters that better suit your needs, these are just defaults:

key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.kafka.connect.storage.StringConverter

comma separated topics to send messages into

topics=DemoPOC

mandatory properties (for a complete properties list see the connector documentation):

The S3 Bucket name or Amazon Resource Name (ARN).

camel.kamelet.aws-s3-source.bucketNameOrArn=arn:aws:s3:::xxxxxxxxxxxxxxxxxxxxx

The AWS region to access.

camel.kamelet.aws-s3-source.region=ap-southeast-1 camel.source.endpoint.autocloseBody=false camel.component.aws-s3.configuration.access-key= camel.component.aws-s3.configuration.secret-key=

oscerd commented 1 year ago

Please use the correct parameters: https://camel.apache.org/camel-kafka-connector/3.18.x/reference/connectors/camel-aws-s3-source-kafka-source-connector.html

You're mixing up configuration from 0.11.x and 3.x. If you are using a ckc version > 1.0 then you should use the camel-kamelets options, like reported in the documentation for the connector.