Currently, nats-kafka uses segmentio/kafka-go to interface with Kafka. Unfortunately, there seems to be a bug that causes unwanted batching of message delivery for users.
After doing some testing, @wallyqs and I discovered that Shopify/sarama doesn't have this issue.
This change replaces segmentio/kafka-go with Shopify/sarama. This fixes the unwanted batching while consuming and also stabilizes 12 out of 17 flaky tests.
The overall approach was to keep much of the existing code, while introducing an interface where we change the underlying library.
Currently, nats-kafka uses segmentio/kafka-go to interface with Kafka. Unfortunately, there seems to be a bug that causes unwanted batching of message delivery for users.
Related issues include:
After doing some testing, @wallyqs and I discovered that Shopify/sarama doesn't have this issue.
This change replaces segmentio/kafka-go with Shopify/sarama. This fixes the unwanted batching while consuming and also stabilizes 12 out of 17 flaky tests.
The overall approach was to keep much of the existing code, while introducing an interface where we change the underlying library.