search

hpgrahsl / kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Apache License 2.0
153 stars 61 forks source link

Error while starting sink connector #108

Closed Dwijad closed 5 years ago

Dwijad commented 5 years ago

Hi I am facing an issue while trying to start a worker/connector(sink). Attaching the startup log as an attachment. I have configured the connector like following:

  1. Downloaded the zip file and after extraction copied all the JARS from lib folder in plugin.path=/u01/cnfkc/confluent-kfk/share/java/kafka-connect-mongodb
  2. The above plugin path is added to the CLASSPATH env variable.
  3. The plugin.path in worker.properities also set to the above path.

When i try to start the connect-distributed using the above properties file, there are lot of class not found errors starting with the following one:

05:35:57.225 [main] INFO  org.reflections.Reflections - Reflections took 96 ms to scan 1 urls, producing 111 keys and 297 values [using 1 cores]
05:35:57.245 [main] WARN  org.reflections.Reflections - could not get type for name javax.mail.Authenticator from any class loader
org.reflections.ReflectionsException: could not get type for name javax.mail.Authenticator

The plugin can also be listed through REST call

[
  {
    "class": "at.grahsl.kafka.connect.mongodb.MongoDbSinkConnector",
    "type": "sink",
    "version": "1.3.1"
  },
..
..

Submitting a connector through REST results in following error which i think cannot connect to MongoDB host although i can ping and connect to the mongodb host/port.

{
  "error_code": 500,
  "message": "IO Error trying to forward REST request: java.net.NoRouteToHostException: No route to host"
}

STartup log

05:36:11.057 [main] WARN  org.reflections.Reflections - could not get type for name org.apache.kafka.clients.MockClient$MockMetadataUpdater from any class loader
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.clients.MockClient$MockMetadataUpdater
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.MockClient$MockMetadataUpdater
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.066 [main] WARN  org.reflections.Reflections - could not get type for name org.jboss.marshalling.ByteInput from any class loader
org.reflections.ReflectionsException: could not get type for name org.jboss.marshalling.ByteInput
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.jboss.marshalling.ByteInput
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.153 [main] WARN  org.reflections.Reflections - could not get type for name org.easymock.IAnswer from any class loader
org.reflections.ReflectionsException: could not get type for name org.easymock.IAnswer
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.easymock.IAnswer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.254 [main] WARN  org.reflections.Reflections - could not get type for name io.netty.channel.SimpleChannelInboundHandler from any class loader
org.reflections.ReflectionsException: could not get type for name io.netty.channel.SimpleChannelInboundHandler
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: io.netty.channel.SimpleChannelInboundHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.294 [main] WARN  org.reflections.Reflections - could not get type for name javax.mail.Authenticator from any class loader
org.reflections.ReflectionsException: could not get type for name javax.mail.Authenticator
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: javax.mail.Authenticator
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.325 [main] WARN  org.reflections.Reflections - could not get type for name org.scalatest.mockito.MockitoSugar from any class loader
org.reflections.ReflectionsException: could not get type for name org.scalatest.mockito.MockitoSugar
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.mockito.MockitoSugar
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.341 [main] WARN  org.reflections.Reflections - could not get type for name org.apache.kafka.test.MockMetricsReporter from any class loader
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.test.MockMetricsReporter
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.test.MockMetricsReporter
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.345 [main] WARN  org.reflections.Reflections - could not get type for name org.osgi.util.tracker.ServiceTracker from any class loader
org.reflections.ReflectionsException: could not get type for name org.osgi.util.tracker.ServiceTracker
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.osgi.util.tracker.ServiceTracker
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.361 [main] WARN  org.reflections.Reflections - could not get type for name io.netty.channel.ChannelInboundHandlerAdapter from any class loader
org.reflections.ReflectionsException: could not get type for name io.netty.channel.ChannelInboundHandlerAdapter
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: io.netty.channel.ChannelInboundHandlerAdapter
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.391 [main] WARN  org.reflections.Reflections - could not get type for name org.eclipse.jetty.npn.NextProtoNego$ServerProvider from any class loader
org.reflections.ReflectionsException: could not get type for name org.eclipse.jetty.npn.NextProtoNego$ServerProvider
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.npn.NextProtoNego$ServerProvider
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.435 [main] WARN  org.reflections.Reflections - could not get type for name javax.jms.MessageListener from any class loader
org.reflections.ReflectionsException: could not get type for name javax.jms.MessageListener
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: javax.jms.MessageListener
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.502 [main] WARN  org.reflections.Reflections - could not get type for name org.eclipse.jetty.jmx.ObjectMBean from any class loader
org.reflections.ReflectionsException: could not get type for name org.eclipse.jetty.jmx.ObjectMBean
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.jmx.ObjectMBean
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 common frames omitted
05:36:11.533 [main] WARN  org.reflections.Reflections - could not get type for name com.google.gson.JsonSerializer from any class loader
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonSerializer
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:400)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:299)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:237)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:185)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:178)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonSerializer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)

It seems all exceptions are related to Reflections.

I have the following JARS in the plugin path

audience-annotations-0.5.0.jar  jackson-core-2.9.8.jar           kafka-connect-mongodb-sink-0.2.2.jar    ojdbc8.jar
bson-3.10.1.jar                 jackson-databind-2.9.8.jar       kafka-schema-registry-client-5.2.0.jar  postgresql-9.4-1206-jdbc41.jar
common-config-5.2.0.jar         jline-0.9.94.jar                 logback-classic-1.2.3.jar               slf4j-api-1.7.25.jar
commons-lang3-3.8.1.jar         jtds-1.3.1.jar                   logback-core-1.2.3.jar                  sqlite-jdbc-3.25.2.jar
common-utils-5.2.0.jar          junit-3.8.1.jar                  mongodb-driver-3.10.1.jar               zkclient-0.10.jar
common-utils-5.2.1.jar          kafka-avro-serializer-5.2.0.jar  mongodb-driver-core-3.10.1.jar          zookeeper-3.4.13.jar
jackson-annotations-2.9.0.jar   kafka-connect-mongodb-1.3.1.jar  netty-3.10.6.Final.jar

Shall i have to include more JAR file in the plugin path ?

hpgrahsl commented 5 years ago

hi @DwijadasDey It's a bit tough to exactly pin point the problem you are facing given the above information...

1) as for the plugin-path this is typically the "main entry" folder where you want the classpath scanning for connector implementations to happen

2) the idea is also to keep every connector inside a separate sub-folder in order to allow for classpath isolation during bootstrap face of connect. here it looks like you are mixing all jars into one folder which belong to different connectors?

The following are comments in a sample connect-distributed.properties file which for instance is taken from the Confluent Platform:

# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins
# (connectors, converters, transformations). The list should consist of top level directories that include 
# any combination of: 
# a) directories immediately containing jars with plugins and their dependencies
# b) uber-jars with plugins and their dependencies
# c) directories immediately containing the package directory structure of classes of plugins and their dependencies

the errors you see may thus result form other connector jars that are also in the plugin-path. the verify this you could remove all the other connectors and only put the mongodb sink connector into your plugin-path and restart connect. you may want to try if if helps to move the complete folder "hpgrahsl-kafka-connect-mongodb-1.3.1" that you get after unzipping into the plugin-path entry folder.

irrespective of this missing dependencies, the error message you get in terms of a 500 "no route to host" is definitely something else and you need to dig deeper and also get more logs which refer to this error.

Dwijad commented 5 years ago

Hi @hpgrahsl I had to configure the connector in the rest of the nodes in the cluster to remove the error messages. Earlier i tried to run the connector by configuring the connector in a single node. Although, I am not sure whether this is the reason for the offending error messages or not.

Earlier even though there were error messages in the startup but the node was able to initialized with other connectors.

Regards

hpgrahsl commented 5 years ago

@DwijadasDey THX for reporting back! So from your answer I conclude that you solved your problem which was that you only installed the sink connector on one of several nodes of your distributed connect deployment. If I'm right I'll close this issue. If not feel free to reopen anytime.