backoff param in idempotent producer has no effect when broker unavailable

cch123 commented 1 year ago

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
6acb276	2.7.0	1.18

Configuration

What configuration values are you using for Sarama and Kafka?

    saramaConfig.Producer.RequiredAcks = sarama.WaitForAll
    saramaConfig.Producer.Idempotent = true
    saramaConfig.Producer.Partitioner = sarama.NewHashPartitioner
    saramaConfig.Producer.Retry.Max = 20
    saramaConfig.Producer.Retry.Backoff = 1 * time.Second // this param has no effect

    saramaConfig.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategyRoundRobin

Logs

When filing an issue please provide logs from Sarama and Kafka if at all possible. You can set sarama.Logger to a log.Logger to capture Sarama debug output.

logs: CLICK ME

``` ```

Problem Description

Our producer is set to idempotent, and AWS Kafka requires updating once a month when the Kafka broker executes a rolling update. However, our producer hangs indefinitely when trying to produce data during this update. Previously, I submitted a PR for this issue (https://github.com/Shopify/sarama/pull/2378) and we patched our producer. However, we have discovered that the 'backoff' parameter does not take effect when the Kafka broker is temporarily unavailable, causing the retry attempts to be quickly exhausted.

github-actions[bot] commented 11 months ago

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

github-actions[bot] commented 7 months ago

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

IBM / sarama