IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.23k stars 1.73k forks source link

Too many goroutines in gopark state #2839

Open ajeyprasad opened 3 months ago

ajeyprasad commented 3 months ago
Description

We have implemented a single Kafka producer using Sarama, which multiple goroutines utilize to produce messages to Kafka. However, we have observed an excessive number of goroutines in the gopark state. What could be the possible cause for this?

Below is the image of goroutine dump:

gopark
Versions
Sarama Kafka Go
v1.40.1 2.8.1 1.18
dnwe commented 3 months ago

So from the displayed goroutine image it looks fairly normal. Your producer is writing to one or more topics totalling 2093 partitions. Sarama dedicates a parititionProducer to each partition. Each of these has a dispatch() loop which is waiting on an input channel to receive messages that you'd like it to send. When that input channel is empty they will block on the chanrecv waiting for input and the Go scheduler will park the goroutine until it needs to be woken when the channel receives input.

github-actions[bot] commented 3 weeks ago

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.