Kafkaproducer: buffer-size and flush-interval precedence

[Ardneliahs]

Hello,

I wanted to understand how buffer-size and flush-interval work with kafka output. I see that flush-interval will flush the buffer at certain interval, what happens if the buffer is full before the interval?

I set flush interval as 30 sec and buffer size as 100, and sent 700 QPS to my DNS server, all of which were forwarded to kafka, if it should discard after 100 messages, how come it sent all 700 per sec, does it flush after reaching 100 messages and it takes precedence over flush-interval?

[dmachard]

Hi,

In your case, the flush interval isn’t triggered because the buffer fills up before the interval timeout. The DNS collector flushes the buffer every time it reaches 100 DNS messages.

The flush interval setting is mainly useful when your traffic is slower.

Best regards, Denis

[Ardneliahs]

Thank you!