Collision with record_modifier plugin

[mirii1994]

Hi, I'm trying to use the record_modifier plugin, along with the rewrite_tag plugin, in the following way:

    <filter **>
      @type record_modifier
      <record>
        type  k8s
        message ${record["log"]}
      </record>
      remove_keys log
    </filter>

    <match kubernetes.var.log.containers.**>
     @type rewrite_tag_filter
     <rule>
       key message
       pattern /\[[A-Z]*\]/
       tag changed
     </rule>
    </match>

What happens is that the logs that match the pattern get the tag changed, but it also makes the message field null. You can see a further example of the way the log appear after the rewrite in this issue. Without the record_modifier plugin, the plugin works (but I have to change the key to log). Can you please advice on how to use those two plugins together?

[vaibhavepatel]

I believe you're missing enable_ruby option in the record_modifier plugin. This may be causing an issue where it's unable to pull the information resulting in null value.

Here's an example provided in the fluentd record_transformer documentation using the ${record[]} option.

<filter foo.bar>
  @type record_transformer
  enable_ruby
  <record>
    avg ${record["total"] / record["count"]}
  </record>
</filter>

[mirii1994]

Hi @vaibhavepatel thanks for replying. I'm using the record_modifier plugin, not record_transformer. It doesn't have the enable_ruby option.

[mirii1994]

Also, i've tried using it with record_transformer with the enable_ruby option, but the issue still repeats itself

[kenhys]

It seems that this issue will be reproduced without rewrite_tag_filter. With remove_keys in filter causes it. :thinking:

<filter **>
  @type record_modifier
  <record>
    type  k8s
    message ${record["log"]}
  </record>
  #remove_keys log
</filter>