hotspot issue with sendMessageAsync on partition level

[raiusa]

We have been using kafka-net code to produce message to kafka broker in production. Recently we found that producer code is producing most of the messages to a specific partition number. Based on my understanding it should be equally distributed across all partitions. Here is snippet of code:

    Dim serverUris = Me.GetServerUris()
   ' build Kafka message(s) and send/write to topic
    Dim options As New KafkaOptions(serverUris)
    Dim router As BrokerRouter
    Try
        router = New BrokerRouter(options)
    Catch ex As Exception
        Dim msg = ex.Message
    End Try

    Dim messages As New List(Of Message)
    For Each messageText As String In serializedMessages
        Dim message As New Message(messageText)
        messages.Add(message)
    Next

    Using client As New Producer(router)
        client.SendMessageAsync(Me.TopicName, messages).Wait()
    End Using

I don't know why it's happening. Please advise me what should I do to fix this issue. It's really critical and breaking our prod env. Waiting for your response. Thanks

[j-alexander]

You'll want to be sure to set the partition key. For instance (in F#):

let message = new KafkaNet.Protocol.Message()
message.Key <- x.Key
message.Value <- x.Value  // already set to messageText by your constructor call

It will hash and mod the key to come up with a partition assignment. As a result the same key always gets the same partition, but on average you get a nice even distribution if your key space is diverse. Try using a primary key or identifier from the subject of your message.

Likely what's happening right now is that it's using the same default key and getting the same partition number.