search

sorenmacbeth / flambo

A Clojure DSL for Apache Spark
Eclipse Public License 1.0
606 stars 84 forks source link

Topic map in flambo.streaming/kafka-stream #91

Closed punit-naik closed 8 years ago

punit-naik commented 8 years ago

I did not quite understand the way topics are passed to the function flambo.streaming/kafka-stream. If I pass a topic map like {"test" 1}, what does the 1 stand for?

joshrotenberg commented 8 years ago

kafka-stream calls Spark's KafkaUtils/createStream. in the topic map, the name is the topic name and the value is the number of partitions to consume, and each partition will be consumed in it's own thread.

More information is available here: http://spark.apache.org/docs/latest/streaming-kafka-integration.html

sorenmacbeth commented 8 years ago

@joshrotenberg thanks for replying!

@punit-naik if that answers your question, would you mind closing the issue, also, in case people haven't see it, there is a flambo-user google group as an alternative place to ask questions.