Small Produced Messages broken up amongst multipel TCP Packets

[stewartboyd119]

Description

When producing messages using confluent-kafka-go, I've noticed that if my message's value (with no key and no headers) is 72 bytes and smaller that it is sent as 1 tcp packet to the broker. Above this it will be 2 or more. I don't see a similar behavior in the sarama library. This manifests as larger latencies. When my producer is deployed to our datacenter and communicates with the kafka broker in our datacenter, the 73 bytes message can take 40 ms to be acked as produced by the broker. The 72 byte message takes single milliseconds.

• 73 bytes -> 2 packets 268 bytes and 188 bytes sent from my local machine to the kafka broker
• 72 bytes -> 1 packets 341 bytes sent from my local machine to the kafka broker

I'm looking for guidance, on why this split my be happening. Is it something in my control (by of config options), and why the sarama producer wouldn't see the same type of packet split.

How to reproduce

Produce a 72 byte message (or 73 byte message with the following snippet) and then monitor outgoing traffic to broker using wireShark (or any network packet inspector).

    var value []byte
    for i := 0; i < valSize; i++ {
        value = append(value, byte(i))
    }

    events := make(chan kafka.Event)
    err := s.kproducer.Produce(&kafka.Message{
        TopicPartition: kafka.TopicPartition{Topic: &topicName, Partition: kafka.PartitionAny},
        Value:          value,
    }, events)

Checklist

Please provide the following information:

• [X] confluent-kafka-go and librdkafka version (LibraryVersion()): v2.0.2
• [ ] Apache Kafka broker version:

• [X] Client configuration: `ConfigMap{ &kafka.ConfigMap{ "bootstrap.servers": bootstrapServers, "client.id": uuid.NewString(), "sasl.username": s.kConfig.SaslUsername, "sasl.password": s.kConfig.SaslPassword, "security.protocol": "SASL_SSL", "sasl.mechanism": "SCRAM-SHA-256", "linger.ms": 0, "batch.num.messages": 1, "socket.send.buffer.bytes": 1000, "socket.keepalive.enable": true, "acks": "all", "enable.idempotence": "true", }

  }`

• [X] Operating system: OsX (13.2.1)
• [ ] Provide client logs (with "debug": ".." as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[milindl]

Hi, this behaviour is rather anomalous. I can't reproduce it with the same settings, the break up into 2 packets happens at very high message sizes, ~30K bytes, in the latest v2.1.1.

Do you see similar behaviour locally, too @stewartboyd119 , or does it happen only on your datacenter machines?

I suspect there might be some network setting external to this library which might cause this, but then it should probably happen for sarama as well.

Does the same behaviour happen when socket.send.buffer.bytes is not set (and left with default value 0)?

Could you share pcaps or similar which we can look at? As well as debug logs for the application, setting "debug" key in the config to "broker".

[milindl]

Closing this - please reopen with the additional information attached and if this still persists. Thanks!