Open pavgup opened 7 years ago
It doesn't effect the behaviour. The problem is fluency and kafka library share same Java's Property. kafka library doesn't ignore non kafka related parameters and this is why this log is generated.
Hmm... I will update the README for it.
Interesting. In that case, @repeatedly, we ran into a problem using that max bytes option.
I can't easily replicate this problem, but here's what took place (and I only know what took place because I found the fix): we ended up running into an issue where fluentd died unexpectedly (still unclear why) and whenever we restarted fluentd the java consumers would grow infinitely large. I attempted to use the fluentd.client.buffer.max.bytes option to limit the size of the buffer the consumer would generate (limiting the overall memory usage), but had no luck. In the end, the thing that solved our problem was deleting ALL of the saved buffer data when the java consumers were exiting. That stopped the RAM usage from growing unbounded.
I know this is the worst kind of issue report -- I don't have the ability to replicate this, but it certainly happened and the fix was removing buffer data. Messy and unclear what was going on.
Any idea why that might have happened? And should this config have limited overall ram usage?
Drat. I'm eager to start using the max buffer size to prevent memory overruns and when I use the following config:
fluentd.client.buffer.max.bytes=128000000
I end up getting this response from the client: