search

manub / scalatest-embedded-kafka

A library that provides an in-memory Kafka instance to run your tests against.
MIT License
294 stars 87 forks source link

Kafka Embedded dependency problems #140

Open guillermoortiz opened 6 years ago

guillermoortiz commented 6 years ago

Hello,

I'm trying to use the version 0.9.0 because I'm using Spark Streaming 2.0.2 and it seems the most compatible version. But this version has some bugs (Java heap problem...) and I need a newer version.

When I try to use a newer version I get some dependencies errors when I start Kafka embedded. The reason seem that my spark-streaming library use kafka-client 0.10.0.1 and if I try to use a newer version from kafka embedded it uses the version 0.10.2.1. If I add this last dependency the code doesn't work because some functions doesn't exist in this new version from kafka client. Is there any way to resolve this issue? The spark version I use: https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10_2.11/2.0.2 I have tried with many differents versions from kafka embedded and only works te version 0.9.0

23:01:05.805 [main] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[main,5,main] died java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:65) ~[idea_rt.jar:na] Caused by: java.lang.NoSuchMethodError: kafka.server.KafkaServer$.$lessinit$greater$default$2()Lorg/apache/kafka/common/utils/Time; at net.manub.embeddedkafka.EmbeddedKafkaSupport$class.startKafka(EmbeddedKafka.scala:467) ~[scalatest-embedded-kafka_2.11-0.14.0.jar:0.14.0] at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:38) ~[scalatest-embedded-kafka_2.11-0.14.0.jar:0.14.0] at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:55) ~[scalatest-embedded-kafka_2.11-0.14.0.jar:0.14.0]

manub commented 6 years ago

Have you tried forking the library and setting the versions to what works for you? If you fork starting from the release you use you could possibly get a matching version.