search

jcustenborder / kafka-connect-spooldir

Kafka Connect connector for reading CSV files into Kafka.
Apache License 2.0
165 stars 124 forks source link

ClassNotFoundException: com.fasterxml.jackson.annotation.JsonKey due to the old jackson-annotations-2.9.0 #156

Closed Hubbitus closed 3 years ago

Hubbitus commented 3 years ago

In ksqlDB (embedded connect) I've do:

CREATE SOURCE CONNECTOR csv_source WITH (
    'connector.class'              = 'com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector',
        'topic'                        = 'test.ksqldb.csv-spooldir',
    'input.path'                    = '/home/pasha/@Projects/@Experiments/ksqlDB/@EPM-SSDL/Network_Device-Location_mapping/Russia/wired/',
    'finished.path'                = '/home/pasha/@Projects/@Experiments/ksqlDB/@EPM-SSDL/Network_Device-Location_mapping/Russia/wired/_csv_processed',
    'error.path'                   = '/home/pasha/@Projects/@Experiments/ksqlDB/@EPM-SSDL/Network_Device-Location_mapping/Russia/wired/_csv_error',
    'input.file.pattern'           = '.*\.csv$',
    'schema.generation.enabled'    = 'true',
    'csv.first.row.as.header'      = 'true',
    'schema.generation.key.fields' = 'Network_Device'
 );
[2020-12-09 03:09:58,505] [org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:192)] ERROR WorkerConnector{id=CSV_SOURCE} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnector:192)
java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey
        at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.hasAsKey(JacksonAnnotationIntrospector.java:1080)
        at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:451)
        at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:378)
        at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:227)
        at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:258)
        at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:391)
        at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:220)
        at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:169)
        at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1473)
        at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1421)
        at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:520)
        at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:798)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
        at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3126)
        at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:388)
        at com.github.jcustenborder.kafka.connect.utils.jackson.SchemaSerializationModule$Serializer.serialize(SchemaSerializationModule.java:183)
        at com.github.jcustenborder.kafka.connect.utils.jackson.SchemaSerializationModule$Serializer.serialize(SchemaSerializationModule.java:179)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
        at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4485)
        at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3740)
        at com.github.jcustenborder.kafka.connect.spooldir.AbstractSchemaGenerator.<clinit>(AbstractSchemaGenerator.java:51)
        at com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector.generator(SpoolDirCsvSourceConnector.java:44)
        at com.github.jcustenborder.kafka.connect.spooldir.AbstractSpoolDirSourceConnector.start(AbstractSpoolDirSourceConnector.java:58)
        at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:184)
        at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:209)
        at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:348)
        at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:331)
        at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:140)
        at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonKey
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 35 more

It looks like updating dependency to jackson-annotations-2.12.0.jar solves the problem.

arijitmazumdar commented 3 years ago

I have faced the similar challenge while implementing CSV file source from control center.

jcustenborder commented 3 years ago

Which version of Kafka Connect are you using?

Hubbitus commented 3 years ago

I use ksqldb embedded kafka connect. How I could obtain its version?

jcustenborder commented 3 years ago

Does the same issue happen with regular connect? I'm wondering if it's an issue with classpath isolation. Regular connect uses plugin path to load connectors. You set plugin.path=<some directory> each plugin under there gets it's own classpath. Problems like this used to happen in the early days of connect when we didn't have this feature.

arijitmazumdar commented 3 years ago

Yes, atleast I am getting this issue with connect with confluent 6.0.1.