help request: multiline settings don't work?

[Dduanduan]

Description

The requirement is to collect multiline log information. like this:

I tried the tail input plugin setting the multiline true or the multiline filter setting the parser. But the problem is these settings don't seem to work. maybe something's wrong,but I have no idea, anybody can give some effective advice. thx!!!

I've already seen at https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-and-containers-v1.8 But! you kown!

then this is my yaml:

The results fell short of expectations:

How did you install fluent operator?

helm upgrade --install fluent-operator --create-namespace -n fluent fluent-operator.tgz

Additional context

version: Kubernetes v1.21.5 Docker 20.10.9 Linux CentOS Linux release 7.9.2009 (Core) Fluent Bit v1.8.11

[github-actions[bot]]

@Dduanduan this issue was automatically closed because it did not follow the issue template

[benjaminhuo]

we support multi-line as I can remember? @wenchajun

[wenchajun]

Yes, you can also take a look at this plugin. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/filter/multiline_types.go You can take a look at the secret and see if the configuration inside is as expected all the time. But I think you can look more at regular expressions for validation. https://mp.weixin.qq.com/s/PI3E5-S0Qhbmn0OgEvYy-Q

[Dduanduan]

@wenchajun first,I tried using the multiline filter and the built-in Java parser worked,The results fell short of expectations:

This should be one record, but it actually returns multiple records. then I want to customize the MULTILINE_PARSER: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing but I can't find the relevant configuration in fluent-operator.

I've tried many times with the tail input plugin regular expression, but the results are not good. Every time I used this site to confirm: https://rubular.com/

So, what should I do?

[wenchajun]

https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/filter/multiline_types.go

You can try this.

[Dduanduan]

@wenchajun How can I customize the MULTILINE_PARSER in fluent-operator.

[wenchajun]

Here is an example https://github.com/fluent/fluent-operator/pull/176

[Dduanduan]

@wenchajun thank you. I saw it: https://github.com/fluent/fluent-operator/pull/176 but it used a built-in plugin,not customize. I kown built-in plugin parsers support: go, java and more. I means [PARSER] and [MULTILINE_PARSER] is different. so, fluent-operator not support customize the MULTILINE_PARSER, right?

[wenchajun]

@wenchajun thank you. I saw it: #176 but it used a built-in plugin,not customize. I kown built-in plugin parsers support: go, java and more. I means [PARSER] and [MULTILINE_PARSER] is different. so, fluent-operator not support customize the MULTILINE_PARSER, right?

Yes, the consideration will be integrated accordingly at a later date.

[richardyc]

Is there any updates to the multiline parser support?