[request] Support spaces in Modify filter keys and values

[jefferbrecht]

I would like to be able to include spaces in Modify conditions, for example:

[FILTER]
    Name modify
    Match *
    Condition Key_value_equals log foo bar
    Add a b

This would add {"a":"b"} to the record if log equals foo bar. Likewise I would want similar behaviour for Key_value_matches, Key_value_does_not_equal, etc.

Currently the above is not a valid config: I get [2021/11/05 19:10:27] [error] [filter:modify:modify.0] Invalid config for Condition.

My only workaround is to translate Key_value_equals log foo bar into the moral equivalent of Key_value_matches log ^foo[^\S\r\n\t\f\v]bar$. I need to be able to use Modify on log records based on arbitrary character patterns, which may include spaces, and having to escape space characters in such a way will make things really complicated.

On a related note, the Grep filter seems to allow spaces without issue, so it's surprising that Modify behaves differently:

[FILTER]
    Name grep
    Match *
    Exclude log foo bar

This successfully loads, and filters out {"log":"foo bar"} as intended.

[nokute78]

We can use Key_value_matches and it supports regex. It means we can use \s as a space.

[INPUT]
    Name Dummy
    Dummy {"log":"foo bar"}

[FILTER]
    Name modify
    Match *
    Condition Key_value_matches log foo\sbar
    Add a b

[OUTPUT]
    Name stdout

$ bin/fluent-bit -c 4286/a.conf 
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/11/07 08:08:08] [ info] [engine] started (pid=4405)
[2021/11/07 08:08:08] [ info] [storage] version=1.1.5, initializing...
[2021/11/07 08:08:08] [ info] [storage] in-memory
[2021/11/07 08:08:08] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/11/07 08:08:08] [ info] [cmetrics] version=0.2.2
[2021/11/07 08:08:08] [ info] [sp] stream processor started
[0] dummy.0: [1636240088.738083636, {"log"=>"foo bar", "a"=>"b"}]
[1] dummy.0: [1636240089.736110456, {"log"=>"foo bar", "a"=>"b"}]
[2] dummy.0: [1636240090.736797318, {"log"=>"foo bar", "a"=>"b"}]
[3] dummy.0: [1636240091.736388108, {"log"=>"foo bar", "a"=>"b"}]
^C[2021/11/07 08:08:13] [engine] caught signal (SIGINT)
[0] dummy.0: [1636240092.736977675, {"log"=>"foo bar", "a"=>"b"}]

[jefferbrecht]

Thanks for looking into this! \s isn't quite equivalent since it matches characters other than ` so we would need a different workaround. We did however identify\x20as an alternative which should be OK for now, although it would still be even better ifModifybehaved consistently withGrep` for space characters.

Edit: We tried surrounding the regex with /.../, e.g. Condition Key_value_matches log /foo bar/ but we still get the same error.

[jefferbrecht]

I found that spaces are also not permitted inside of record accessors:

[FILTER]
    Name modify
    Match *
    Condition Key_value_equals $log['foo bar'] abcdef
    Add a b

This causes [2021/11/12 20:09:48] [error] [filter:modify:modify.0] Invalid config for Condition. Unlike the original problem with spaces in a regex, there is no workaround for spaces in record accessors since record accessors do not use escape sequences.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[franciscovalentecastro]

The change https://github.com/fluent/fluent-bit/pull/6387 committed into fluent-bit master addresses the request for values within quotes ", though AFAIU it doesn't address the problem with keys.