Modify filter no longer supports Condition key

[dfishburn]

Bug Report

Describe the bug A filter modify with a condition clause works in 1.8.15 but fails in 2.0.8

The error message:

fluent-bit [2023/01/11 19:28:55] [debug] [filter:modify:filter-iots-fluentbit] Initialized modify filter with 0 conditions and 2 rules
fluent-bit [2023/01/11 19:28:55] [error] [config] parser: unknown configuration property 'condition'. The following properties are allowed: Key_Name, Parser, Preserve_Key, Reserv
fluent-bit [2023/01/11 19:28:55] [ help] try the command: /fluent-bit/bin/fluent-bit -F parser -h

Removing the filter modify with condition results in successful execution.

To Reproduce The error stems from FILTERs modify which uses the Condition clause:

[FILTER]
    Name                     modify
    Alias                    filter-iots-haproxy-set-fields
    Match                    iots-haproxy*
    Condition                Key_Does_Not_Exist iot_haproxy_message
    Copy                     log iot_haproxy_message
    Add                      iot_component haproxy
    Add                      iot_fileset_module iot
    Add                      iot_haproxy_level INFO
    Set                      iot_fileset_name haproxy

The FluentBit 2.x documentation still lists the Condition syntax

• Steps to reproduce the problem:
  • Simply create a filter modify with a condition

Expected behavior This is a regression from 1.x. Same syntax should work as documented.

Your Environment

• Version used: 2.0.8
• Configuration:
  • Running in Kubernetes

[patrick-stephens]

Please can you provide the full config.

I think there must be something else going on with your configuration as this works fine for me:

$ cat fluent-bit.conf 
[INPUT]
  Name dummy
  Tag test

[FILTER]
  Name Modify 
  Match *
  Condition Key_Does_Not_Exist iot_haproxy_message
  Copy                     log iot_haproxy_message
  Add                      iot_component haproxy
  Add                      iot_fileset_module iot
  Add                      iot_haproxy_level INFO
  Set                      iot_fileset_name haproxy

[OUTPUT]
  Name null
  Match *

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit
Fluent Bit v2.0.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/01/12 09:34:29] [ info] [fluent bit] version=2.0.8, commit=9444fdc5ee, pid=1
[2023/01/12 09:34:29] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/01/12 09:34:29] [ info] [cmetrics] version=0.5.8
[2023/01/12 09:34:29] [ info] [ctraces ] version=0.2.7
[2023/01/12 09:34:29] [ info] [input:dummy:dummy.0] initializing
[2023/01/12 09:34:29] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/01/12 09:34:29] [ info] [output:null:null.0] worker #0 started
[2023/01/12 09:34:29] [ info] [sp] stream processor started

Plus the error references the parser filter rather than the modify one. If I deliberately introduce an incorrect condition (condition abc) then I correctly get an error message about the modify filter:

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit
Fluent Bit v2.0.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/01/12 09:33:31] [ info] [fluent bit] version=2.0.8, commit=9444fdc5ee, pid=1
[2023/01/12 09:33:31] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/01/12 09:33:31] [ info] [cmetrics] version=0.5.8
[2023/01/12 09:33:31] [ info] [ctraces ] version=0.2.7
[2023/01/12 09:33:31] [ info] [input:dummy:dummy.0] initializing
[2023/01/12 09:33:31] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/01/12 09:33:31] [error] [filter:modify:modify.0] Invalid config for condition : abc

I would be curious if you have a parser filter defined in your config.

[nokute78]

@dfishburn

fluent-bit [2023/01/11 19:28:55] [error] [config] parser: unknown configuration property 'condition'. The following properties are allowed: Key_Name, Parser, Preserve_Key, Reserv

Did you use filter_parser ? These allowed properties are for filter_parser, not for filter_modify. https://docs.fluentbit.io/manual/pipeline/filters/parser

[dfishburn]

I will narrow down the case to see. I had 6 filters has I outlined in my original post, and when I removed ALL of them, fluent-bit started up successfully. This time, I will remove one at a time and see where I get.

[patrick-stephens]

And to be clear: it may be that in older versions you could pass in invalid config and it did not fail but now it will. That is expected behaviour - I personally would prefer an explicit failure over hidden warnings when you make a typo in the configuration parameter for example that meant your config is not what you think it is or requested (another common example is using the docs for the Grafana Loki custom plugin to configure the Fluent Bit Loki plugin which is quite different).

[dfishburn]

Sorry for the time you have wasted colleagues. I did find a separate case where I was using a parser and not a modify clause. I appreciate the responsiveness of this group. David