IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.56k stars 1.76k forks source link

Why does the pprof analysis show that github.com/IBM/sarama.(*partitionProducer).dispatch has a large number of coroutines enabled and suspended #2980

Open Aliuyanfeng opened 2 months ago

Aliuyanfeng commented 2 months ago
Description

as shown below,I am in the process of producing messages and find that a large number of coroutines are started

image

Versions
Sarama Kafka Go
v1.43.2 V2_5_0_0 1.20
Configuration
func (p *KFKPool) newConnection() (sarama.SyncProducer, error) {
    config := sarama.NewConfig()
    config.Version = sarama.V2_5_0_0
    config.Producer.RequiredAcks = sarama.WaitForLocal
    config.Producer.Partitioner = sarama.NewRandomPartitioner
    config.Producer.Return.Successes = true
    config.Producer.Timeout = time.Second * 5
    config.Producer.Retry.Max = 0
    config.Net.WriteTimeout = time.Second * 3
    client, err := sarama.NewSyncProducer([]string{p.host}, config)
    if err != nil {
        return nil, err
    }
    return client, nil
}

Looking forward to receiving your reply, thank you!🤣

puellanivis commented 2 months ago

This is expected. There’s quite a few different goroutines that get started per partition. If you have enough partitions, then this number can explode pretty fast. However, most of them are parked at any given time, so their impact on performance is minimal.