search

fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.85k stars 1.58k forks source link

Character limit for Kafka broker in Fluentbit configuration. #707

Open AkshayDubey29 opened 6 years ago

AkshayDubey29 commented 6 years ago

Can one tell me how do I increase the character limit in broker configuration for kafka

StevenACoffman commented 6 years ago

Kafka brokers have their own configuration, independent of fluent-bit. Fluent-bit kafka output uses a kafka client library called librdkafka that can produce messages to kafka brokers. You can pass any configuration option that librdkafka supports.

If you look at the librdkafka configuration documentation you might be thinking of message.max.bytes which you can configure with a prefix of rdkafka.

For instance look at this fluent-bit kafka output configuration with these librdkafka options configured

Does that answer your question?

StevenACoffman commented 6 years ago

I noticed that this feature is undocumentated, so I made a quick PR.

edsiper commented 6 years ago

internal note: Monkey rconf interface buffer string size is 1KB, it needs to be expanded.