search

startappdev / kafka-connect-mongodb

MongoDB sink connector for Kafka Connect
Apache License 2.0
13 stars 9 forks source link

Does not work: Too many errors on Java 1.8.0_131-b11 #3

Closed inancgumus closed 7 years ago

inancgumus commented 7 years ago

I followed the README, when I try to register the connector, the following errors appear. Where am I doing wrong?

I copied the jar into the project folder and added to CLASSPATH. Zookeeper, kafka and even schema-registry is actively listening.

echo $CLASSPATH
/Users/inanc/dev/br/integration-test/kafkasinktest/lib/kafka-connect-mongodb-assembly-1.0.jar

# start the connect
connect-distributed.sh ./connect-distributed.properties

# register the connector
# after this command, connect log displays errors as in the below header: 
# ERROR MESSAGES WHEN I TRY TO REGISTER
curl -X POST -H "Content-Type: application/json" --data @./mongo_connector_configs.json http://localhost:8083/connectors

connect-distributed.properties:

bootstrap.servers=localhost:9092

group.id=testGroup

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=false
value.converter.schemas.enable=false

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

offset.storage.topic=connectoffsets

offset.flush.interval.ms=10000
config.storage.topic=connectconfigs

# I NEEDED TO ADD THIS OR IT DISPLAYS ERRORS:
# I also created connectstatus topic for this
status.storage.topic=connectstatus

mongo_connector_configs.json

I even tried to create the mongo testdb with some bogus data.

{
       "name":"mongo-connector-testTopic",
       "config" :{
               "connector.class":"com.startapp.data.MongoSinkConnector",
               "tasks.max":"5",
               "db.host":"localhost",
               "db.port":"27017",
               "db.name":"testdb",
               "db.collections":"testcollection",
               "write.batch.enabled":"true",
               "write.batch.size":"200",
               "connect.use_schema":"false",
               "topics":"testTopic"
       }
}

ERROR MESSAGES WHEN I TRY TO REGISTER:

[2017-05-06 13:51:25,123] INFO Kafka version : 0.10.2.1 (org.apache.kafka.common.utils.AppInfoParser:83)
[2017-05-06 13:51:25,123] INFO Kafka commitId : e89bffd6b2eff799 (org.apache.kafka.common.utils.AppInfoParser:84)
[2017-05-06 13:51:25,123] INFO ConsumerConfig values:
    auto.commit.interval.ms = 5000
    auto.offset.reset = earliest
    bootstrap.servers = [localhost:9092]
    check.crcs = true
    client.id =
    connections.max.idle.ms = 540000
    enable.auto.commit = false
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = testGroup
    heartbeat.interval.ms = 3000
    interceptor.classes = null
    key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.ms = 50
    request.timeout.ms = 305000
    retry.backoff.ms = 100
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
 (org.apache.kafka.clients.consumer.ConsumerConfig:180)
[2017-05-06 13:51:25,125] WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'offset.flush.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,125] WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,126] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,126] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2017-05-06 13:51:25,126] INFO Kafka version : 0.10.2.1 (org.apache.kafka.common.utils.AppInfoParser:83)
[2017-05-06 13:51:25,126] INFO Kafka commitId : e89bffd6b2eff799 (org.apache.kafka.common.utils.AppInfoParser:84)
[2017-05-06 13:51:25,133] INFO Discovered coordinator 192.168.99.1:9092 (id: 2147483647 rack: null) for group testGroup. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:586)
[2017-05-06 13:51:25,136] INFO Finished reading KafkaBasedLog for topic connectconfigs (org.apache.kafka.connect.util.KafkaBasedLog:146)
[2017-05-06 13:51:25,136] INFO Started KafkaBasedLog for topic connectconfigs (org.apache.kafka.connect.util.KafkaBasedLog:148)
[2017-05-06 13:51:25,136] INFO Started KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore:248)
[2017-05-06 13:51:25,136] INFO Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:195)
[2017-05-06 13:51:25,138] INFO Discovered coordinator 192.168.99.1:9092 (id: 2147483647 rack: null) for group testGroup. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:586)
[2017-05-06 13:51:25,140] INFO (Re-)joining group testGroup (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:420)
[2017-05-06 13:51:25,148] INFO Successfully joined group testGroup with generation 7 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:388)
[2017-05-06 13:51:25,149] INFO Joined group and got assignment: Assignment{error=0, leader='connect-1-e78cdc33-aefd-4572-a42d-e09f63ce02e3', leaderUrl='http://192.168.1.26:8083/', offset=-1, connectorIds=[], taskIds=[]} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1151)
[2017-05-06 13:51:25,149] INFO Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:814)
[2017-05-06 13:51:25,149] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:824)
May 06, 2017 1:51:25 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2017-05-06 13:51:25,414] INFO Started o.e.j.s.ServletContextHandler@3e10dc6{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)
[2017-05-06 13:51:25,431] INFO Started ServerConnector@320de594{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)
[2017-05-06 13:51:25,431] INFO Started @1017ms (org.eclipse.jetty.server.Server:379)
[2017-05-06 13:51:25,431] INFO REST server listening at http://192.168.1.26:8083/, advertising URL http://192.168.1.26:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150)
[2017-05-06 13:51:25,431] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)
[2017-05-06 13:51:26,674] INFO Reflections took 1506 ms to scan 63 urls, producing 3991 keys and 24266 values  (org.reflections.Reflections:229)
[2017-05-06 13:51:28,967] WARN  (org.eclipse.jetty.servlet.ServletHandler:620)
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
    at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:509)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
    ... 23 more
Caused by: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.apache.kafka.connect.connector.Connector.validate(Connector.java:133)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:254)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:508)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:505)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:248)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:198)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more
[2017-05-06 13:51:28,969] INFO 0:0:0:0:0:0:0:1 - - [06/May/2017:10:51:28 +0000] "POST /connectors HTTP/1.1" 500 294  72 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2017-05-06 13:51:28,969] WARN /connectors (org.eclipse.jetty.server.HttpChannel:396)
javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:130)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
    ... 10 more
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
    at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:509)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
    ... 23 more
Caused by: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef;
    at org.apache.kafka.connect.connector.Connector.validate(Connector.java:133)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:254)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:508)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:505)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:248)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:198)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more
[2017-05-06 13:51:28,971] WARN Could not send response error 500: javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.AbstractMethodError: org.apache.kafka.connect.connector.Connector.config()Lorg/apache/kafka/common/config/ConfigDef; (org.eclipse.jetty.server.HttpChannel:482)
inancgumus commented 7 years ago

Interesting thing is: When I try to query the registered plugins, MongoSinkConnector appears there!

$ curl -sX GET http://localhost:8083/connector-plugins/ | jq
[
  {
    "class": "com.startapp.data.MongoSinkConnector"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSinkConnector"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSourceConnector"
  }
]
RazOmessi commented 7 years ago

Hi,

I see you are using Kafka 10, The jar is for Kafka 9, You can build it from source for Kafka 10.

Any way i will build it myself for 10 and upload the new jar.

Raz