skroutz / rafka

Kafka proxy with a simple API, speaking the Redis protocol
https://engineering.skroutz.gr/blog/kafka-rails-integration/
GNU General Public License v3.0
8 stars 0 forks source link

producer: Consume event channel in a goroutine #37

Closed agis closed 7 years ago

agis commented 7 years ago

The librdkafka producer events channel should be consumed prior to calling Flush(), otherwise we'll hit the flush timeout and outstanding events will be reported by Flush() (for more info see https://github.com/confluentinc/confluent-kafka-go/blob/v0.11.0/kafka/producer.go#L195-L215)

Therefore, we have to constantly consume events by the Events() channel until the producer is Close()ed, therefore its events channel is closed. We introduced a new goroutine, consumeDeliveries that does just that, consumes events and logs any failures.

Some refactorings were also performed:

Fixes #33