Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

Is it possible that one spout subscribes multiple topics from kafka #161

Closed meowoodie closed 9 years ago

meowoodie commented 9 years ago

I have a question that is it possible that one spout subscribes multiple topics from kafka. For an instance, a spout named "test_spout" receives streaming data from two topics in kafka. The yaml file may looks like this:

topology:
    - spout:
        name: kafka_user_raw_data
        type: kafka
        options:
            #module: streaming_topology.user_behavior_spout
            # The Kafka topic to stream from.
            # Required.
            topic:
              - topic1
              - topic2

But it couldn't work. Could you please tell me how to achieve this?

mzbyszynski commented 9 years ago

As far as I know it is not possible, but you can have multiple kafka spouts in your topology, and have them all be consumed by the same bolt. Would that do the job?

meowoodie commented 9 years ago

@mzbyszynski Thank you, I did this with multiple spouts indeed.