search

nats-io / nats-kafka

NATS to Kafka Bridging
Apache License 2.0
131 stars 32 forks source link

Replace segmentio/kafka-go with Shopify/sarama #25

Closed variadico closed 3 years ago

variadico commented 3 years ago

Currently, nats-kafka uses segmentio/kafka-go to interface with Kafka. Unfortunately, there seems to be a bug that causes unwanted batching of message delivery for users.

Related issues include:

After doing some testing, @wallyqs and I discovered that Shopify/sarama doesn't have this issue.

This change replaces segmentio/kafka-go with Shopify/sarama. This fixes the unwanted batching while consuming and also stabilizes 12 out of 17 flaky tests.

The overall approach was to keep much of the existing code, while introducing an interface where we change the underlying library.

variadico commented 3 years ago

All right. Changes: