rakutentech / kafka-firehose-nozzle

Forward logs from the Cloud Foundry Firehose to Apache Kafka
MIT License
13 stars 8 forks source link

Add buffer on nozzle #13

Closed tcnksm closed 7 years ago

tcnksm commented 7 years ago

When sarama input buffer is full, whole process (consuming & producing) is blocked. This PR fix that problem. It prevents blocking on sarama partition dispatcher. Drop message when buffer is full.

Not only that adding a new buffer layer on nozzle side (subInput). So that we don't lose logs.