MongoDB Sink - Exception UnmodifiableIterator

[psmod2]

Hi,

I've download the Mongo Sink (for Kafka version 1.0) from here: https://lenses.stream/connectors/sink/mongo.html

I've extracted the JAR out and created a docker image with the base FROM confluentinc/cp-kafka-connect:4.0.0 with the single jar from the MongoSink in the folder: /usr/share/java/.

In my docker-compose I then set the plugin path (as per Confluent documentation) using: CONNECT_PLUGIN_PATH: /usr/share/java/

However I keep getting the error:

[2018-04-25 03:33:54,806] INFO Loading plugin from: /usr/share/java/kafka-connect-mongodb-1.0.0-1.0.0-all.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-04-25 03:34:19,526] INFO Reflections took 24380 ms to scan 1 urls, producing 6252 keys and 49763 values  (org.reflections.Reflections)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
    at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
    at org.reflections.Reflections.<init>(Reflections.java:126)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)

Any ideas what is wrong?

Are there other dependencies/JARs i need to have?

Thanks.

[Antwnis]

The stack-trace java.lang.NoSuchMethodError means indeed that you are missing some JARs. The rest of the exception com.google.common.collect.Sets indicates that the Guava library is not available

[psmod2]

@Antwnis - thanks - however is there a list of dependencies I need (as I'm just using the connector) and not Stream Reactor as a whole? And if its just Guava - is there a specific location (i.e. Landoop I can get the JAR)

Thanks for your help

[psmod2]

Also one other thing (just for my curiosity) - I download the StreamReactor and looked in the libs folder - guava wasn't there either? Is this expected? Thanks.

[andrewstevenson]

@psmod2 Guava should be there. We create fatJars, it's required by some of the drivers like HBase, I don't know about Mongo of the top of my head. We also have prebuilt dockers for each connector.

Are you using the fatJar?

[Antwnis]

@psmod2 I will close this ticket - but please re-open if the following does not address your issue

gradle :kafka-connect-mongodb:shadowJar

ls -la kafka-connect-mongodb/build/libs

This will generate a FAT JAR ( 60 MBytes JAR ) that contains all the dependencies.

You probably executed the :build task - that builds only the stream-reactor lib ( a 130 KB lib ). With that you would have to collect and provide the dependencies as a manual step - and that's what you probably missed and got the guava missing JAR exception

I hope the above resolves this ticket

[psmod2]

@Antwnis @andrewstevenson Thanks - however I download the JAR from this location: https://lenses.stream/connectors/sink/mongo.html It was actually a 58MB JAR so i presumed it contained libs?

[psmod2]

@Antwnis Running the Gradle build as described above, unfortunately I still get the same error:

[2018-04-26 05:30:05,687] INFO Loading plugin from: /usr/share/java/kafka-connect-mongodb-1.1.0-SNAPSHOT-1.0.0-all.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2018-04-26 05:30:36,755] INFO Reflections took 30526 ms to scan 1 urls, producing 6252 keys and 49776 values  (org.reflections.Reflections)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
    at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
    at org.reflections.Reflections.<init>(Reflections.java:126)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:156)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)

Also FYI - in the stream-reactor-master build.gradle file I had to update classpath 'com.github.maiflai:gradle-scalatest:0.18' to classpath 'com.github.maiflai:gradle-scalatest:0.19' for the build to succeed.

And I didnt see any mention of Guava in those .build files - should it not be there?

[psmod2]

@Antwnis - Also FYI - I'm placing the JARs within a docker image with the following as the base:

FROM confluentinc/cp-kafka-connect:4.0.0

And within its etc/kafka-connect/jars/ folder I noticed guava-18.0.jar. Is this the issue - should it be version 20? If so I presume my only option is to use a more recent version of the docker image FROM confluentinc/cp-kafka-connect:4.0.0 ?

[andrewstevenson]

@psmod2 Have you tried our standalone docker with mongodb or fast-data-dev?

[andrewstevenson]

closing due to inactivity