Lift Operation in Nest Filter Not Working as Expected

[i386kernel]

Bug Report

Describe the bug I've encountered an issue with the lift operation in the nest filter not functioning as expected. Despite configuring the lift operation to move nested JSON data to the root level, the data remains nested under the original key.

Configuration

Below is my configuration for Fluent Bit, with sensitive data obscured:

[PARSER]
Name cri
Format regex
Regex .?\s(?<log>{.})$
Time_Key time
Time_Format %Y/%m/%d %H:%M:%S

[INPUT]
Name tail
Path /var/log/containers/component-dataplane*.log
Tag nginx.access
Mem_Buf_Limit 50Mb
Parser cri
Skip_Long_Lines On

[FILTER]
Name nest
Match nginx.access
Operation lift
Nested_under log
Add_prefix ""

Steps to Reproduce

1. Configure Fluent Bit with the above configuration.
2. Observe the output in Splunk or any log management tool.

Expected Behavior

The lift operation should promote the nested JSON under log to the top level of the log structure.

Actual Behavior

However, the logs still appear with the nested log structure when viewed in Splunk. Here's an example of how logs are currently formatted with dummy data:

{
"appid": "XXXXXX",
"cloud_environment": "dummy_cloud",
"colour": "dummy_colour",
"component_appid": "XXXXXX",
"environment": "dummy_env",
"log": {"user_id":null,"messageId":null,"swimlane":null,"x_consumer_id":null,"originatingSystemId":null,"remote_address":"dummy_address","host":"dummy_host","x_wdp_request_id":null,"status":200,"appCorrelationId":null,"body_bytes_sent":41,"user_agent":"dummy_agent","request_time":1,"time":"dummy_time","request":"GET /dummy/path HTTP/1.1"}
}

Additional Context

• Fluent Bit version: fluent-bit:2.2.2
• Environment: [Azure Kubernetes Service, 1.28.3]

Any guidance on resolving this issue or would be greatly appreciated.

Thank you for your support.

[nokute78]

Could you try to remove Add_prefix if it is not used ?

Add_prefix of filter_nest is not working on Windows. https://github.com/fluent/fluent-bit/issues/8137

Following configuration is a minimum conf.

[INPUT]
    Name   dummy
    dummy {"log":{"a":"b", "b":"c"}}

[FILTER]
    Name      nest
    Match     *
    operation lift
    nested_under log
#    add_prefix ""

[OUTPUT]
    Name           stdout
    Match          *

[i386kernel]

Sorry, for the delayed response, I was able to fix this using 'Parser' Filter, like so...

Name tail
    Path /var/log/containers/${HOSTNAME}_component_consul-dataplane*.log
    Tag nginx.access
    Mem_Buf_Limit 50Mb
    Parser cri
    Skip_Long_Lines On

[OUTPUT]
    name splunk
    match nginx.access
    host hec.npr.lip.srv.westpac.com.au
    port 443
    splunk_token <Your_Splunk_Token_Here>
    event_host ${NODENAME}
    event_source /logs/${COMPONENT_NAME}/${HOSTNAME}/nginx-access.log
    event_sourcetype consul-dataplane
    event_index event_index
    tls on
    tls.verify off

[FILTER]
    Name parser
    Match nginx.access
    Key_Name message
    Parser nginx_access
    Reserve_Data False

parsers_json.conf: |
  [PARSER]
    Name nginx_access
    Format json
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep On
  [PARSER]
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep On