consumer read from specific partition

[joshjdevl]

Hi,

Suppose I have N partitions. I would like to have X different consumer threads ( X < N) read from a specified set of partitions. How can I achieve this?

Thanks

[sclasen]

The AkkaConsumer uses actors instead of threads under the covers, but to have N consumer streams, you just need pass streams = N when creating your AkkaConsumerProps.

This number gets passed through to the underlying high-level kafka consumer, which then returns N streams.

You would typically start N threads against those streams, but with akka-kafka there are N StreamFSM actors created instead.

[sclasen]

Or I may have misunderstood your question. If you want to control consuming from a subset of the available partitions of a given topic, this is not supported atm.

[joshjdevl]

Yes, I would like to control consuming from a subset of the available partitions of a given topic

[sclasen]

As you saw on the kafka mailing list, you would need to use the simple consumer for this.

akka-kafka is based on the high level consumer and as such cant currently support this.