search

awslabs / kinesis-kafka-connector

kinesis-kafka-connector is connector based on Kafka Connect to publish messages to Amazon Kinesis streams or Amazon Kinesis Firehose.
Apache License 2.0
153 stars 91 forks source link

Unable to start the connector #10

Closed anshulagl closed 7 years ago

anshulagl commented 7 years ago

I am running the connector as a service on kubernetes. My docker file looks something like this:

From wurstmeister/kafka

COPY ./jars /opt/kafka/libs

CMD ["sh", "-c", "/opt/kafka/bin/connect-standalone.sh /etc/config/worker.properties /etc/config/kinesis-firehose-kafka-connector.properties"]

Kafka-Connect Version: 0.11.0.1

Error Seen:

Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
    at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
    at org.reflections.Reflections.<init>(Reflections.java:126)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:221)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:159)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:63)

So, I tried using Version 0.9.0.0, it also threw an exception:

Error Logs:

2017-10-05T20:48:59.668122443Z [2017-10-05 20:48:59,667] INFO Sink task org.apache.kafka.connect.runtime.WorkerSinkTask@6cfd4f44 finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:155)
2017-10-05T20:48:59.962278903Z [2017-10-05 20:48:59,960] ERROR Task analytics-kafka-connector-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask:302)
2017-10-05T20:48:59.965636578Z [2017-10-05 20:48:59,961] ERROR Task is being killed and will not recover until manually restarted: (org.apache.kafka.connect.runtime.WorkerSinkTask:303)
2017-10-05T20:48:59.965662749Z java.lang.NullPointerException
2017-10-05T20:48:59.965673047Z  at com.amazon.kinesis.kafka.DataUtility.parseValue(DataUtility.java:25)
2017-10-05T20:48:59.965681012Z  at com.amazon.kinesis.kafka.DataUtility.createRecord(DataUtility.java:101)
2017-10-05T20:48:59.965689179Z  at com.amazon.kinesis.kafka.FirehoseSinkTask.putRecordsInBatch(FirehoseSinkTask.java:137)
2017-10-05T20:48:59.965696319Z  at com.amazon.kinesis.kafka.FirehoseSinkTask.put(FirehoseSinkTask.java:55)
2017-10-05T20:48:59.965712345Z  at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:280)
2017-10-05T20:48:59.965739587Z  at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176)
2017-10-05T20:48:59.965750069Z  at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
2017-10-05T20:48:59.965757396Z  at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
2017-10-05T20:48:59.965764006Z  at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
2017-10-05T20:48:59.967391023Z [2017-10-05 20:48:59,965] ERROR Thread WorkerSinkTask-analytics-kafka-connector-0 exiting with uncaught exception:  (org.apache.kafka.connect.util.ShutdownableThread:84)
2017-10-05T20:48:59.967438126Z org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
2017-10-05T20:48:59.967449020Z  at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:304)
2017-10-05T20:48:59.967456903Z  at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176)
2017-10-05T20:48:59.967464509Z  at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
2017-10-05T20:48:59.967471901Z  at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
2017-10-05T20:48:59.967479363Z  at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
2017-10-05T20:48:59.968196274Z Exception in thread "WorkerSinkTask-analytics-kafka-connector-0" org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
2017-10-05T20:48:59.968217098Z  at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:304)
2017-10-05T20:48:59.968225928Z  at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176)
2017-10-05T20:48:59.968235976Z  at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
2017-10-05T20:48:59.968243474Z  at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
2017-10-05T20:48:59.968250793Z  at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
nehalmehta commented 7 years ago

Hi @anshul35

Can you provide more details --

1) How have you build Kinesis Kafka Connector? 2) Which Kafka Version are you using? 3) Which Kafka Connect API are you using?

Thanks, Nehal

anshulagl commented 7 years ago

Hi,

  1. Built using "mvn package" on mac-osx
  2. Kafka version 0.0.10
  3. Tested with both Kafka connect version 0.0.9 and 0.0.11
anshulagl commented 7 years ago

Changed the key/value converter to StringConverter in worker.properties and it worked. Earlier, was using JsonConverter