search

upyun / cafex

Cafex is a pure Elixir implementation of Kafka client
Other
70 stars 7 forks source link

Consumer with multiple topics #3

Open mbuhot opened 8 years ago

mbuhot commented 8 years ago

Hi and thanks for creating Cafex!

I'd like to have a consumer receive input from two or more topics to perform some stream joining, similar to how a samza job might be structured: (https://samza.apache.org/learn/documentation/0.9/container/state-management.html)

It looks like the underlying Kafka Fetch protocol supports multiple topics (https://github.com/upyun/cafex/blob/master/lib/cafex/protocol/fetch.ex#L10), but it is not surfaced in the Cafex consumer configuration API?

In the meantime, is the best alternative to start a consumer for each topic, and forward the messages to my own GenServer, which can maintain some state and produce the outputs?

Thanks again!

belltoy commented 8 years ago

Yes, I'm afraid you're right.

According to the Kafka Protocol Guild, batching is a very significant performance win. Most of the APIs support batching across multiple topics. Cafex will support it, in one consumer.