nats-io / nats-kafka

NATS to Kafka Bridging
Apache License 2.0
131 stars 32 forks source link

Fixed Kafka bug with consumer groups and timeouts during repartitioning #80

Closed jonchammer closed 1 year ago

jonchammer commented 1 year ago

Background In https://github.com/Shopify/sarama/issues/1192, a user reported seeing read tcp <ip>:63133-><ip>:9092: i/o timeout errors when consuming Kafka messages with a consumer group during rebalancing. As a response, Sarama updated their documentation and examples to indicate that the ConsumeClaim function should explicitly check for a completed Context (see: https://github.com/Shopify/sarama/pull/2240). It's possible that we may have run into the same problem when using nats-kafka, as we recently saw very similar errors in a production installation.

Description With this PR, we're updating nats-kafka's Kafka Consumer to align with the updated usage advice from the Sarama developers (https://github.com/Shopify/sarama/blob/main/examples/consumergroup/main.go).