fluent / fluent-operator

Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Apache License 2.0
587 stars 250 forks source link

Fluent-bit: Add log_to_metrics filter plugin #1285

Closed jcdauchy-moodys closed 3 months ago

jcdauchy-moodys commented 3 months ago

Is your feature request related to a problem? Please describe.

Really useful plugin to extract metrics from logs:

https://docs.fluentbit.io/manual/pipeline/filters/log_to_metrics

Describe the solution you'd like

It could be used with RemoteWrite plugin or other output plugins.

Thanks

Additional context

No response

Athishpranav2003 commented 3 months ago

@jcdauchy-moodys https://github.com/fluent/fluent-operator/blob/0f632a794feb743b890ba1cde13145ccb1781dd3/docs/best-practice/custom-plugin.md this might be helpful for your case

@cosmo0920 will it be useful to add the interface for this plugin? Adding the plugin to codebase is simply adding the interface but not sure if its indeed necessary tho. If its needed maybe i can do this one quick

cosmo0920 commented 3 months ago

To use fluent bit plugins, we might need to create a map (or interface?) to use actual parameters for plugins. For log_to_metrics filter plugin, we defined the config map: https://github.com/fluent/fluent-bit/blob/master/plugins/filter_log_to_metrics/log_to_metrics.c#L965-L1052

For adding to build metrics type of pipelines, this plugin would be useful because it's only one plugin to bypass/translate logs type of pipelines into metrics type of pipelines.

Athishpranav2003 commented 3 months ago

Oh ok Then i will go ahead and work on this then