search

treasure-data / kafka-fluentd-consumer

Kafka Consumer for Fluentd
32 stars 18 forks source link

Property fluentd.client.buffer.max.bytes is not valid #25

Open pavgup opened 7 years ago

pavgup commented 7 years ago

Drat. I'm eager to start using the max buffer size to prevent memory overruns and when I use the following config:

fluentd.client.buffer.max.bytes=128000000

I end up getting this response from the client:

2017-08-30 22:04:32,612 [main] (kafka.utils.Logging$class:68) INFO  Verifying properties
2017-08-30 22:04:33,220 [main] (kafka.utils.Logging$class:68) INFO  Property auto.commit.enable is overridden to false
2017-08-30 22:04:33,275 [main] (kafka.utils.Logging$class:68) INFO  Property auto.commit.interval.ms is overridden to 60000
2017-08-30 22:04:33,276 [main] (kafka.utils.Logging$class:68) INFO  Property auto.offset.reset is overridden to largest
2017-08-30 22:04:33,276 [main] (kafka.utils.Logging$class:68) INFO  Property consumer.timeout.ms is overridden to 10
2017-08-30 22:04:33,277 [main] (kafka.utils.Logging$class:83) WARN  Property fluentd.client.buffer.max.bytes is not valid
repeatedly commented 7 years ago

It doesn't effect the behaviour. The problem is fluency and kafka library share same Java's Property. kafka library doesn't ignore non kafka related parameters and this is why this log is generated.

Hmm... I will update the README for it.

pavgup commented 7 years ago

Interesting. In that case, @repeatedly, we ran into a problem using that max bytes option.

I can't easily replicate this problem, but here's what took place (and I only know what took place because I found the fix): we ended up running into an issue where fluentd died unexpectedly (still unclear why) and whenever we restarted fluentd the java consumers would grow infinitely large. I attempted to use the fluentd.client.buffer.max.bytes option to limit the size of the buffer the consumer would generate (limiting the overall memory usage), but had no luck. In the end, the thing that solved our problem was deleting ALL of the saved buffer data when the java consumers were exiting. That stopped the RAM usage from growing unbounded.

I know this is the worst kind of issue report -- I don't have the ability to replicate this, but it certainly happened and the fix was removing buffer data. Messy and unclear what was going on.

Any idea why that might have happened? And should this config have limited overall ram usage?