hash_value_field support (from fluentd) for fluent-bit's filter_parsers

[kfdm]

Is your feature request related to a problem? Please describe.

Using filter_parser you can parse a field into a new value. For example

[FILTER]
    Name parser
    Match systemd.json
    Parser systemd_json
    Key_name MESSAGE
    Reserve_Data On

Allows me to take a MESSAGE field from some incoming systemd (journald) logs, and parse them. One minor issue, is that it all gets merged down into the original hash value

# Input
{"MESSAGE": '{"some":"value"}}
# Output
{"MESSAGE": '{"some":"value"}, "some":value}

Fluentd's version of filter_parser supports a hash_value_field field, that lets us control where this goes.

# Input
{"MESSAGE": '{"some":"value"}}
# Output
{"MESSAGE": '{"some":"value"}, "parsed": {"some":value}}

What would be involved in allowing fluent-bit's filter_parser to support a similar hash_value_field target, to avoid a case where something parsed out might overwrite another field

I think it's not really possible to use filter_lua to parse it, and I think there's no way to have filter_record_modifer to make a copy either. I'm trying to simplify a few things to avoid deploying fluentd, so I'm hoping there might be a way to handle this within fluent-bit alone.

[Athishpranav2003]

I guess i can work on this feature. I am trying to refactor some code in msgpack itself https://github.com/msgpack/msgpack-c/pull/1135. Once that PR gets accepted i will create this PR for the approval

[Athishpranav2003]

Raised PR #9243 for the latest lib

[Athishpranav2003]

@kfdm if possible you can also look at the PR

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.