encrypt / suppress sensitive data in the log

[samar-elsayed]

Describe the bug

I am using KAFKA_OPTS as a var for rdkafka options and it is working fine but the issue is I need to either :

• suppress the config from the log while logging other INFO
• make the var appear as it is in the log

To Reproduce

create var KAFKA_OPTS = "sasl.mechanism: SCRAM-SHA-512,security.protocol: sasl_ssl,sasl.username: user,sasl.password: pass"

and then use it in the config as shown in Your Configuration section

Expected behavior

the config shouldn't appear in the log or at least the var value will be shown as encrypted ****

Your Environment

- Fluentd version: 1.14.0
- fluent-plugin-kafka version: 0.17.5
- ruby-kafka version: 1.5.0
- rdkafka: 0.12.0
- Operating system: Alpine Linux v3.13
- Kernel version: 5.15.58-flatcar

Your Configuration

<source>
   @type tail
   path /logs/quarkus.log
   tag file.all
   <parse>
       @type regexp
       expression /^(?<datetime>[0-9- :,]+) (?<host>[0-9a-zA-Z\-\.\+]+) (?<processname>.+?) (?<loglevel>.+) +\[(?<logger>[a-zA-Z-.]+?)\] \((?<thread>.+?)\) (?<logmessage>.+)$/
   </parse>
</source>

<match file.all>
   @type rewrite_tag_filter
   <rule>
       key logmessage
       pattern /ULFFRecord\:\ (?<ulffrecord>.+)$/
       tag file.ulff
   </rule>
   <rule>
       key logmessage
       pattern /./
       tag file.generic
   </rule>
</match>

<filter file.ulff>
   @type parser
   key_name logmessage
   <parse>
       @type regexp
       expression /^ULFFRecord\:\ (?<ulffrecord>.+)$/
   </parse>
</filter>

<filter file.ulff>
   @type parser
   format json
   key_name ulffrecord
</filter>

<match file.ulff>
   @type rdkafka2
   brokers "<broker>"
   get_kafka_client_log true
   default_topic ulff
   flush_interval 3s
   use_event_time true
   **rdkafka_options "#{ENV["KAFKA_OPTS"]}"**
   <buffer>
       flush_mode interval
       flush_interval 2s
   </buffer>
   <format>
     @type "json"
   </format>
</match>

<match file.generic>
   @type rdkafka2
   enable_ruby
   brokers "<broker>"
   get_kafka_client_log true
   default_topic custom
   use_event_time true
   rdkafka_options "#{ENV["KAFKA_OPTS"]}"
   <buffer>
       flush_mode interval
       flush_interval 2s
   </buffer>
   <format>
     @type "json"
   </format>
</match>

Your Error Log

the var value is logged along with other info

Additional context

No response

[ashie]

It's already fixed in master branch (by #452) but not released yet. You can use password parameter instead of placing it in rdkafka_options. It will be masked in the log.

I'll release it in the near future to ship it in the next major version of td-agent.

[ashie]

It's already fixed in master branch (by https://github.com/fluent/fluent-plugin-kafka/pull/452) but not released yet.

Sorry, I misunderstood it. It's already released. https://github.com/fluent/fluent-plugin-kafka/blob/6f22abb7f5c2a3f0627c997574404703dfc6c9e1/ChangeLog#L6 Please use password parameter with v0.18.0 or later.