search

llofberg / kafka-connect-rest

Kafka Connect REST connector
Apache License 2.0
108 stars 55 forks source link

{id=RestSinkConnector-0} Task is being killed and will not recover until manually restarted #4

Closed braghettos closed 6 years ago

braghettos commented 6 years ago

Hi @llofberg ,

I'm trying to use your sink connector but I'm facing this issue and I'm unable to find out what is the issue:

echo "===> Launching ... "
+ echo '===> Launching ... '
exec /etc/confluent/docker/launch
+ exec /etc/confluent/docker/launch
[2018-06-14 14:21:28,150] INFO Loading plugin from: /jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:28,151] DEBUG Loading plugin urls: [file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar] (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar (org.reflections.Reflections)
[2018-06-14 14:21:28,200] DEBUG [Thread[org.reflections-scanner-0,5,main]] scanning file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar (org.reflections.Reflections)
[2018-06-14 14:21:28,253] DEBUG could not scan file META-INF/MANIFEST.MF in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,277] DEBUG could not scan file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
org.reflections.ReflectionsException: could not create class object from file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.xml
[2018-06-14 14:21:28,277] DEBUG could not scan file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
org.reflections.ReflectionsException: could not create class object from file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.properties
[2018-06-14 14:21:28,283] DEBUG could not scan file META-INF/maven/org.slf4j/slf4j-api/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,284] DEBUG could not scan file META-INF/maven/org.slf4j/slf4j-api/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,284] DEBUG could not scan file META-INF/NOTICE.txt in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,289] DEBUG could not scan file META-INF/LICENSE.txt in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,311] DEBUG could not scan file META-INF/maven/org.apache.commons/commons-compress/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,331] DEBUG could not scan file META-INF/maven/org.apache.commons/commons-compress/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,334] DEBUG could not scan file META-INF/LICENSE in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,335] DEBUG could not scan file META-INF/NOTICE in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,347] DEBUG could not scan file org/apache/velocity/runtime/defaults/velocity.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,351] DEBUG could not scan file org/apache/velocity/runtime/defaults/directive.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,364] DEBUG could not scan file META-INF/DEPENDENCIES in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,366] DEBUG could not scan file META-INF/maven/org.apache.velocity/velocity-engine-core/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,371] DEBUG could not scan file META-INF/maven/org.apache.velocity/velocity-engine-core/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,373] DEBUG could not scan file META-INF/maven/commons-io/commons-io/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,373] DEBUG could not scan file META-INF/maven/commons-io/commons-io/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,380] DEBUG could not scan file META-INF/maven/org.apache.commons/commons-lang3/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,400] DEBUG could not scan file META-INF/maven/org.apache.commons/commons-lang3/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,414] WARN could not get type for name com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter
[2018-06-14 14:21:28,420] WARN could not get type for name com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter
[2018-06-14 14:21:28,429] WARN could not get type for name com.tm.kafka.connect.rest.selector.TopicSelector from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.selector.TopicSelector
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.selector.TopicSelector
[2018-06-14 14:21:28,460] INFO Registered loader: PluginClassLoader{pluginLocation=file:/jars/kafka-connect-rest-1.0-SNAPSHOT-shaded.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:28,460] INFO Added plugin 'com.tm.kafka.connect.rest.RestSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:28,461] INFO Added plugin 'com.tm.kafka.connect.rest.RestSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:28,470] INFO Loading plugin from: /jars/kafka-connect-rest-1.0-SNAPSHOT.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:28,471] DEBUG Loading plugin urls: [file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar] (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar (org.reflections.Reflections)
[2018-06-14 14:21:28,471] DEBUG [Thread[org.reflections-scanner-0,5,main]] scanning file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar (org.reflections.Reflections)
[2018-06-14 14:21:28,471] DEBUG could not scan file META-INF/MANIFEST.MF in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar with scanner SubTypesScanner (org.reflections.Reflections)
[2018-06-14 14:21:28,474] DEBUG could not scan file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.xml in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar with scanner SubTypesScanner (org.reflections.Reflections)
org.reflections.ReflectionsException: could not create class object from file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.xml
[2018-06-14 14:21:28,474] DEBUG could not scan file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.properties in url file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar with scanner SubTypesScanner (org.reflections.Reflections)
org.reflections.ReflectionsException: could not create class object from file META-INF/maven/com.tm.kafka/kafka-connect-rest/pom.properties
[2018-06-14 14:21:28,475] WARN could not get type for name com.tm.kafka.connect.rest.selector.TopicSelector from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.selector.TopicSelector
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.selector.TopicSelector
[2018-06-14 14:21:28,475] WARN could not get type for name com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter
[2018-06-14 14:21:28,475] WARN could not get type for name com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter
[2018-06-14 14:21:28,483] INFO Registered loader: PluginClassLoader{pluginLocation=file:/jars/kafka-connect-rest-1.0-SNAPSHOT.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:28,493] WARN could not get type for name com.tm.kafka.connect.rest.selector.TopicSelector from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.selector.TopicSelector
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.selector.TopicSelector
[2018-06-14 14:21:28,493] WARN could not get type for name com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.converter.SinkRecordToPayloadConverter
[2018-06-14 14:21:28,493] WARN could not get type for name com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter
Caused by: java.lang.ClassNotFoundException: com.tm.kafka.connect.rest.converter.PayloadToSourceRecordConverter
        rest.advertised.host.name = kafka-connect
        rest.advertised.listener = null
        rest.advertised.port = null
        rest.host.name = kafka-connect
        rest.port = 8083
[2018-06-14 14:21:30,375] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:30,375] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:30,375] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:30,594] INFO Added connector for http://kafka-connect:8083 (org.apache.kafka.connect.runtime.rest.RestServer)
[2018-06-14 14:21:30,699] INFO Advertised URI: http://kafka-connect:8083/ (org.apache.kafka.connect.runtime.rest.RestServer)
[2018-06-14 14:21:30,866] INFO Starting REST server (org.apache.kafka.connect.runtime.rest.RestServer)
[2018-06-14 14:21:30,869] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:30,869] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:30,869] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:31,001] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,001] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,001] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,021] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,021] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,021] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,047] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:31,047] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:31,047] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:31,161] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,161] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,161] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,164] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,164] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,164] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,222] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:31,222] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2018-06-14 14:21:31,222] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)
Jun 14, 2018 2:21:31 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2018-06-14 14:21:31,311] INFO Advertised URI: http://kafka-connect:8083/ (org.apache.kafka.connect.runtime.rest.RestServer)
[2018-06-14 14:21:31,311] INFO REST server listening at http://kafka-connect:8083/, advertising URL http://kafka-connect:8083/ (org.apache.kafka.connect.runtime.rest.RestServer)
[2018-06-14 14:21:31,331] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,331] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,331] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2018-06-14 14:21:31,333] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,333] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,333] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-06-14 14:21:31,347] DEBUG Updating configuration for connector RestSinkConnector configuration: {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, res .sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.storage.KafkaConfigBackingStore)
[2018-06-14 14:21:31,347] DEBUG Storing new config for task RestSinkConnector-0 this will wait for a commit message before the new config will take effect. New config: {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, task.class=com.tm.kafka.connect.rest.RestSinkTask, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, res .sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.storage.KafkaConfigBackingStore)
        connector.class = com.tm.kafka.connect.rest.RestSinkConnector
        connector.class = com.tm.kafka.connect.rest.RestSinkConnector
        connector.class = com.tm.kafka.connect.rest.RestSinkConnector
        connector.class = com.tm.kafka.connect.rest.RestSinkConnector
[2018-06-14 14:21:34,859] DEBUG Getting plugin class loader for connector: 'com.tm.kafka.connect.rest.RestSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:34,859] INFO Creating connector RestSinkConnector of type com.tm.kafka.connect.rest.RestSinkConnector (org.apache.kafka.connect.runtime.Worker)
        task.class = class com.tm.kafka.connect.rest.RestSinkTask
[2018-06-14 14:21:34,860] INFO Instantiated task RestSinkConnector-0 with version 1.0-SNAPSHOT of type com.tm.kafka.connect.rest.RestSinkTask (org.apache.kafka.connect.runtime.Worker)
[2018-06-14 14:21:34,862] INFO Instantiated connector RestSinkConnector with version 1.0-SNAPSHOT of type class com.tm.kafka.connect.rest.RestSinkConnector (org.apache.kafka.connect.runtime.Worker)
[2018-06-14 14:21:34,862] DEBUG Getting plugin class loader for connector: 'com.tm.kafka.connect.rest.RestSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-06-14 14:21:34,862] DEBUG WorkerConnector{id=RestSinkConnector} Initializing connector RestSinkConnector with config {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.runtime.WorkerConnector)
        rest.sink.method = POST
        rest.sink.payload.converter.class = class com.tm.kafka.connect.rest.converter.JsonPayloadConverter
        rest.sink.payload.converter.schema = true
        rest.sink.properties = [Content-Type:application/json]
        rest.sink.retry.backoff.ms = 5000
        rest.sink.url = http://webservice:8080/count
        rest.sink.velocity.template = rest.vm
 (com.tm.kafka.connect.rest.RestSinkConnectorConfig)
[2018-06-14 14:21:34,890] DEBUG Getting plugin class loader for connector: 'com.tm.kafka.connect.rest.RestSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
        connector.class = com.tm.kafka.connect.rest.RestSinkConnector
        connector.class = com.tm.kafka.connect.rest.RestSinkConnector
[2018-06-14 14:21:34,890] DEBUG Getting plugin class loader for connector: 'com.tm.kafka.connect.rest.RestSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
        rest.sink.method = POST
        rest.sink.payload.converter.class = class com.tm.kafka.connect.rest.converter.JsonPayloadConverter
        rest.sink.payload.converter.schema = true
        rest.sink.properties = [Content-Type:application/json]
        rest.sink.retry.backoff.ms = 5000
        rest.sink.url = http://webservice:8080/count
        rest.sink.velocity.template = rest.vm
 (com.tm.kafka.connect.rest.RestSinkConnectorConfig)
[2018-06-14 14:21:37,961] ERROR WorkerSinkTask{id=RestSinkConnector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2018-06-14 14:21:37,961] DEBUG Stopping sink task, setting client to null (com.tm.kafka.connect.rest.RestSinkTask)

Could you please help me understanding what's wrong in my configuration?

Many thanks,

Diego

llofberg commented 6 years ago

Unfortunately the info in the log doesn't seem to help much.

A couple of things come mind.

Did you create the topic "wimt_collaudo" before running the connector? Is "http://webservice:8080/count" running and reachabble in your environment?

I'd need more information about how you run it to find the issue?

Thanks,

braghettos commented 6 years ago

HI @llofberg, yes the topic was created before running the connector. Even the webservice is reacheable:

root@eb4631b752d0:/# curl -X POST "http://webservice:8080/count" {"timestamp":"2018-06-14T13:33:55.004+0000","status":400,"error":"Bad Request","message":"Required request body is missing: public void hello.GreetingController.count(java.lang.Object)","path":"/count"}

This is the error log:

echo "===> Launching ... "
+ echo '===> Launching ... '
exec /etc/confluent/docker/launch
+ exec /etc/confluent/docker/launch
[2018-06-14 14:21:28,430] DEBUG expanded subtype org.apache.kafka.connect.connector.Task -> org.apache.kafka.connect.sink.SinkTask (org.reflections.Reflections)
[2018-06-14 14:21:28,430] DEBUG expanded subtype org.apache.kafka.connect.connector.Connector -> org.apache.kafka.connect.sink.SinkConnector (org.reflections.Reflections)
[2018-06-14 14:21:28,475] DEBUG expanded subtype org.apache.kafka.connect.connector.Task -> org.apache.kafka.connect.sink.SinkTask (org.reflections.Reflections)
[2018-06-14 14:21:28,475] DEBUG expanded subtype org.apache.kafka.connect.connector.Connector -> org.apache.kafka.connect.sink.SinkConnector (org.reflections.Reflections)
[2018-06-14 14:21:28,494] DEBUG expanded subtype org.apache.kafka.connect.connector.Task -> org.apache.kafka.connect.sink.SinkTask (org.reflections.Reflections)
[2018-06-14 14:21:28,494] DEBUG expanded subtype org.apache.kafka.connect.connector.Connector -> org.apache.kafka.connect.sink.SinkConnector (org.reflections.Reflections)
Jun 14, 2018 2:21:31 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2018-06-14 14:21:31,347] DEBUG Updating configuration for connector RestSinkConnector configuration: {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.storage.KafkaConfigBackingStore)
[2018-06-14 14:21:31,347] DEBUG Storing new config for task RestSinkConnector-0 this will wait for a commit message before the new config will take effect. New config: {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, task.class=com.tm.kafka.connect.rest.RestSinkTask, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.storage.KafkaConfigBackingStore)
[2018-06-14 14:21:34,862] DEBUG WorkerConnector{id=RestSinkConnector} Initializing connector RestSinkConnector with config {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.runtime.WorkerConnector)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-read (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-send (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-active-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;partition-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-seq-number (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion-skip (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;put-batch-time (org.apache.kafka.common.metrics.Metrics)
        rest.sink.method = POST
        rest.sink.payload.converter.class = class com.tm.kafka.connect.rest.converter.JsonPayloadConverter
        rest.sink.payload.converter.schema = true
        rest.sink.properties = [Content-Type:application/json]
        rest.sink.retry.backoff.ms = 5000
        rest.sink.url = http://webservice:8080/count
        rest.sink.velocity.template = rest.vm
        rest.sink.method = POST
        rest.sink.payload.converter.class = class com.tm.kafka.connect.rest.converter.JsonPayloadConverter
        rest.sink.payload.converter.schema = true
        rest.sink.properties = [Content-Type:application/json]
        rest.sink.retry.backoff.ms = 5000
        rest.sink.url = http://webservice:8080/count
        rest.sink.velocity.template = rest.vm
[2018-06-14 14:21:37,961] DEBUG Stopping sink task, setting client to null (com.tm.kafka.connect.rest.RestSinkTask)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;partition-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion-skip (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-send (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-active-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-seq-number (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;put-batch-time (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-read (org.apache.kafka.common.metrics.Metrics)
braghettos commented 6 years ago

This is how I launch the cointainer:

docker run -d \
  --name=kafka-connect \
  --net=confluent \
  -e CONNECT_PRODUCER_INTERCEPTOR_CLASSES=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor \
  -e CONNECT_CONSUMER_INTERCEPTOR_CLASSES=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor \
  -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
  -e CONNECT_REST_PORT=8083 \
  -e CONNECT_GROUP_ID="docker-connect-group" \
  -e CONNECT_CONFIG_STORAGE_TOPIC="docker-connect-configs" \
  -e CONNECT_OFFSET_STORAGE_TOPIC="docker-connect-offsets" \
  -e CONNECT_STATUS_STORAGE_TOPIC="docker-connect-status" \
  -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_REST_ADVERTISED_HOST_NAME="kafka-connect" \
  -e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
  -e CONNECT_LOG4J_LOGGERS=org.reflections=DEBUG \
  -e CONNECT_PLUGIN_PATH=/jars \
  -e CONNECT_REST_HOST_NAME="kafka-connect" \
  -v /home/U370780/kafka-connect-rest/target:/jars \
  -v /home/U370780/kafka-connect-rest/examples/spring/config:/config \
  confluentinc/cp-kafka-connect:4.1.0
llofberg commented 6 years ago

Did you have matching Kafka version (4.1.0) in pom.xml when you built the connector?

braghettos commented 6 years ago

Hi @llofberg , in the default pom.xml the kafka version is 1.1.0 and confluent.version is 4.1.1, which property should I change?

1.1.0 4.1.1
braghettos commented 6 years ago

I found the error in the task of the connector:

root@03a30e469d6d:/# curl 10.0.13.86:8083/connectors/RestSinkConnector/status {"name":"RestSinkConnector","connector":{"state":"RUNNING","worker_id":"kafka-connect:8083"},"tasks":[{"state":"FAILED","trace":"org.apache.kafka.connect.errors.DataException: JsonConverter with schemas.enable requires \"schema\" and \"payload\" fields and may not contain additional fields. If you are trying to deserialize plain JSON data, set schemas.enable=false in your converter configuration.\n\tat org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:338)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:467)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:301)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:205)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:173)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n","id":0,"worker_id":"kafka-connect:8083"}],"type":"sink"}