search

glazkovalex / Rebus.Kafka

Apache Kafka transport for Rebus
MIT License
16 stars 6 forks source link

Working with Partitions #2

Closed arielmoraes closed 4 years ago

arielmoraes commented 4 years ago

How configure the Transport to work with Partitions? Are there any special configurations?

glazkovalex commented 4 years ago

How configure the Transport to work with Partitions? Are there any special configurations?

Hello, @arielmoraes ! Thank you for your question. I don't quite understand your question, as transport is already working with Kafka's partitions. They are created automatically by the Kafka server, depending on the number of bus clients (customers). Specify please in what scenario, in what case of application of the Rebus bus you want to adjust partitions?

If you want to increase the default number of log partitions wpar topic in order to create a worker queue, then this is done not in the bus transport, but on the Kafka server. By increasing the parameter "num.partitions" in the file: kafka/config/server.properties.

The default number of log partitions per topic. More partitions allow greater parallelism for consumption, but this will also result in more files across the brokers. num.partitions=2

Read more Kafka documentation.

By setting the "num.partitions=2 " will allow you to see how the scaling of two workers (Scaleout.Consumers) performing tasks created by one producer (Scaleout.Producer).

glazkovalex commented 4 years ago

Apparently, the request is allowed.