Protobuf producer serialization slow

[bichselb]

Description

I am using the protobuf producer outlined here. Unfortunately, its serialization is so slow that it a bottleneck in my whole system. Specifically, it takes 1.6s to serialize 1 million messages, even though pure protobuf can serialize the same 1 million messages in 100ms.

How can I speed up serialization to avoid this 15x overhead? I would like to avoid parallelization.

How to reproduce

I put my whole experimental setup here: https://github.com/bichselb/confluent-kafka-go-performance/tree/serialization-slow

My serialization is a straight-forward adaptation of the protobuf producer example, except that I enabled CacheSchemas (without this, the code is another order of magnitude slower):

func serializeWithSchemaregistry(nMessages int) {
    registry, err := schemaregistry.NewClient(schemaregistry.NewConfig("http://schema-registry:8081"))
    if err != nil {
        fmt.Printf("Error creating schema registry client...\n")
        panic(err)
    }
    defer registry.Close()

    serializer, err := protobuf.NewSerializer(
        registry,
        serde.ValueSerde,
        &protobuf.SerializerConfig{
            SerializerConfig: *serde.NewSerializerConfig(),
            CacheSchemas: true,  // big difference for performance
        },
    )
    if err != nil {
        fmt.Printf("Error creating serializer...\n")
        panic(err)
    }
    defer serializer.Close()

    start := time.Now()
    topic := "mytopic"
    for i := 0; i < nMessages; i++ {
        message := pb.MyMessage{
            A: int32(i),
            B: int32(i),
        }

        payload, err := serializer.Serialize(topic, &message)
        if err != nil {
            fmt.Printf("Error serializing message...\n")
            panic(err)
        }
        // Pseudo-use of payload to prevent compiler optimizations
        _ = copy(payload, payload)
    }
    fmt.Printf("Protobuf+schemaregistry: Serialized %d messages in %v\n", nMessages, time.Since(start))
}

Checklist

Please provide the following information:

• [x] confluent-kafka-go version: v2.5.4
• [x] Apache Kafka broker version: confluentinc/cp-kafka:7.7.1
• [x] Client configuration: Not needed to showcase this issue
• [x] Operating system: Docker compose on an Ubuntu system
• [x] Provide client logs: See https://github.com/bichselb/confluent-kafka-go-performance/blob/serialization-slow/results.log
• [x] Provide broker log excerpts: See https://github.com/bichselb/confluent-kafka-go-performance/blob/serialization-slow/results.log