consumer group messages not getting delivered

alexec commented 3 years ago

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
v1.29.1	2.13-2.7.0	1.16

Configuration

What configuration values are you using for Sarama and Kafka?


    config.Consumer.MaxProcessingTime = 10 * time.Second
    config.Consumer.Offsets.AutoCommit.Enable = false
    if x.StartOffset == "First" {
        config.Consumer.Offsets.Initial = sarama.OffsetOldest
    }

Logs

When filing an issue please provide logs from Sarama and Kafka if at all possible. You can set sarama.Logger to a log.Logger to capture Sarama debug output.

logs: CLICK ME

``` ```

Problem Description

Argo Dataflow runs on Kubernetes, so it can be SIGTERM at any time. When this happens we gracefully shutdown by closing our consumer group and client connections. When we reconnect, we start processing messages again, but about 30% of the time, we don't seem to continue consuming all remaining messages, the consumer group's offset seem to get stuck at an offset older than the latest.

pachmu commented 3 years ago

I have the same issue using goka (sarama based lib, v1.27.0) After the pod is restarted, sometimes one or more partitions of the topic it reads get stuck. Consuming continues only after 7 days when kafka removes old messages from the topic. @alexec have you found some workaround?

alexec commented 3 years ago

Kind of. We now use Confluent Client (i.e. librdkafka)

github-actions[bot] commented 1 year ago

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

IBM / sarama