search

dibbhatt / kafka-spark-consumer

High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.
Apache License 2.0
635 stars 318 forks source link

How to use the application to handle multi topic #40

Closed zdxMike closed 5 years ago

dibbhatt commented 7 years ago

Yes, it a open issue still. This consumer can not handle multi topic. But if this a urgent need from community , I can bring this in.

zdxMike commented 7 years ago

Thank you for your answer.

I am looking forward to you can bring this in.

konstantin-niko commented 7 years ago

Is there a workaround for this issue? I tried creating separate streams and then use union to merge them but did not work. Also could you, please, outline the scope of the work needed to support multi topic?

ritesh-dineout commented 6 years ago

Yes this is a major drawback for this consumer. Spark out-of-box kafka streaming supports this.

Kindly add this at the earliest.

dibbhatt commented 6 years ago

Hi, I can understand this. I will work on it to add multi topic support. Will keep you posted.

yangcong3643 commented 5 years ago

Does it come true now?

jfgosselin commented 5 years ago

I also need multi topic support.

dibbhatt commented 5 years ago

Hi All. Just released Version 2.1.0 that supports Kafka Multi Topic fetch. Example is mentioned in Readme file. Kindly let me know how this goes .

Sorry I keep you all waiting for such a long time for this, this was over due for quite sometime .

dibbhatt commented 5 years ago

Sample code you can see here :

https://github.com/dibbhatt/kafka-spark-consumer#handling-multi-topic

dibbhatt commented 5 years ago

Hi, let me know if anyone tried multi topic feature . Then I can close this ticket

dibbhatt commented 5 years ago

closing this ticket.