fluent / fluent-plugin-kafka

Kafka input and output plugin for Fluentd
Other
303 stars 178 forks source link

out_rdkafka2: Handle some of the exceptions as unrecoverable errors #510

Closed cosmo0920 closed 4 months ago

cosmo0920 commented 5 months ago

This is because msg_size_too_large and topic_authorization_failed error codes should be treated as an unrecoverable error. In Fluentd, we need to mark as unrecoverable with raise Fluent::Unrecoverable. But current implementation does not handle them as unrecoverable errors.

cosmo0920 commented 5 months ago

FYI: we use flog to generate arbitrary length of logs. In this case, we use flog -b 1024000 -n 1 to create too huge line to consume by out_rdkafka2 plugin.

ashie commented 4 months ago

Thanks!