Does Fluent-bit have log aggregation or log concat functions?

[xiao-Shu-Tiao]

Is your feature request related to a problem? Please describe. As we all know, we can compress the log before writing to kafka. If only one message is compressed, the compression efficiency is low. If multiple messages can be aggregated together, compressed and sent to kafka, the message will be compressed. The efficiency will increase and can take up less storage space, so I hope to have a filter that can aggregate several messages before sending to kafka. Special note is that this logic is different from the logic of multiline merged logs. Multiple log fragments merged by multiline belong to one complete log, such as JAVA stack log. This requirement requires merging several complete logs into one aggregated log.

Describe the solution you'd like I want this filter to have the following parameters: (1) key: The key for part of multiline log (2) separator: The separator of lines (3) n_lines: Number of aggregated lines (4) flush_interval: The number of seconds after which the last received event log will be flushed. If specified 0, wait for next line forever An example configuration is as follows:

[FILTER]
     Name concat
     Match*
     Key source
     separator "\n"
     n_lines 100
     flush_interval 30s

Sample data is as follows: {"log":"a","source":"/path/to/data1"} {"log":"b","source":"/path/to/data1"} {"log":"c","source":"/path/to/data2"} After passing this filter, logs with the same key 'source' will be aggregated into one log according to the separator. {"log":"a\nb","source":"/path/to/data1"} {"log":"c","source":"/path/to/data2"}

This concat filter is very popular in the fluentd community. The address is #https://github.com/fluent-plugins-nursery/fluent-plugin-concat, and it is also an officially recommended solution by fluentd, so I hope the fluent-bit community can also Thank you for having such a filter!

Describe alternatives you've considered If you can provide parameters in kafka_output to support log aggregation, it is also possible.It would be better if this solution could support regular expression aggregation of multiple lines like the plugin of the fluentd community(https://github.com/fluent-plugins-nursery/fluent-plugin-concat?tab=readme-ov-file#usage).

Additional context I hope the community can design and support this requirement, and this feature can promote the improvement of fluent-bit's functions.

[xiao-Shu-Tiao]

Is anyone can help :)

[leoswaldo]

Maybe try asking in the Slack channel, https://www.launchpass.com/fluent-all

[xiao-Shu-Tiao]

Maybe try asking in the Slack channel, https://www.launchpass.com/fluent-all Thanks :)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.