search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
90 stars 869 forks source link

Custom Partitioners #587

Closed mhowlett closed 3 years ago

mhowlett commented 6 years ago

this is a frequently requested feature. librdkafka now provides a built in partitioner that is compatible with the java client partitioner: murmur2_random, but there are other use cases for a custom partitioner. we plan to implement this by adding a ConfigureTopics method to the producer which will have the restriction that it must be called before the topic is first used. This method will take a config dictionary, where one of the properties that can be set is a partitioner delegate.

gunjanarora commented 6 years ago

When can we expect this feature to be available?

mhowlett commented 6 years ago

after the 1.0 API is stable, probably towards the end of the year.

rjrizzuto commented 5 years ago

I also would like this feature. I have an existing application that does streams processing using NEsper. I would like to switch to Kafka for the pub/sub bus. I think mapping event categories to topics makes the most sense for that, as opposed to putting all categories in a single topic. The issue I have is that I need to ensure that partitions in each topic are mapped to the same consumer by key so that the consumer can still join streams based on that key. I essentially need the same partitionor that Kafka Streams uses.

rjrizzuto commented 5 years ago

I see that 1.1.0 is out in Nuget. Is there any ETA on custom partitioners yet?

rohits-dev commented 5 years ago

any news on this feature?

rjrizzuto commented 5 years ago

.Net is the red haired step child still.

Berezhnyk commented 5 years ago

I am waiting for this feature too...

mhowlett commented 5 years ago

It's lower down the priority list since it's easy to work around (just explicitly specify the partition on produce, determined via a class called MyFancyPartitioner if you want). We do plan to implement this still, and it's not hard so it'll get done.

mhowlett commented 5 years ago

it's actually a roadmap item in our Q1 planning, so I just discovered, (but don't rely on that, it'll be something that'll get bumped if other things slip).

rjrizzuto commented 5 years ago

Setting the partition in the producer doesn't seem to allow for a custom rebalance algorithm in the comsumers, like a custom partitioner would.

DobrovAlexey commented 3 years ago

Hi @mhowlett, is there any news on this feature?

mhowlett commented 3 years ago

custom partitioners are in 1.6.2

lambdakris commented 3 years ago

Hi @mhowlett is there documentation for this feature yet? If not, is there a comparable example in java or another lang that I could take a look at and try to adapt?

mhowlett commented 3 years ago

have a look at the integration tests for example use: https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/test/Confluent.Kafka.IntegrationTests/Tests/Producer_CustomPartitioner.cs