Support for `format_firstline` directive

[ryhamz]

Right now I have a working solution using the regular fluentd multiline parser.

 format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}/
 format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<thread>.*?)\] (?<severity>[A-z]*) (?<logger>.*?) - (?<message>.*)/

That is, I know my log blocks start with TIMESTAMP_ISO8601 and everything after it should be included until we see another timestamp to start a line.

Does grok-parser have support for these semantics?

As far as I can tell, multiline_start_regexp /^[^\s]/ tells the plugin that lines to be combined start with whitespace. Some of the stack traces I'm looking to pick up are not indented though.

Am I able to use negative lookahead in multiline_start_regexp in order to say "my multilines start with anything except a timestamp"?

i.e. multiline_start_regexp (?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})

[okkez]

Does grok-parser have support for these semantics?

Do you mean that multiline_start_regexp accept grok like following?

multiline_start_regexp %{TIMESTAMP_ISO8601}

If so, you cannot use grok in multiline_start_regexp.

Am I able to use negative lookahead in multiline_start_regexp in order to say "my multilines start with anything except a timestamp"?

Yes. You must use /(?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})/ form.

[ryhamz]

@okkez Thanks for the reply! I wasn't attempting to use Grok in multiline_start_regexp. I was wondering if the grok plugin had a similar parameter to format_firstline.

I have now attempted the following configuration:

<parse>
 @type multiline_grok
    grok_pattern ^%{TIMESTAMP_ISO8601:timestamp} %{DATA:thread} %{LOGLEVEL:severity}? %{DATA:logger} - %{GREEDYDATA:message}$
    multiline_start_regexp /(?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})/
    time_format %Y-%m-%d %H:%M:%S
  </parse>

My logs in stackdriver do not end up combined though.

Example in Stackdriver

Every part of the stack trace becomes its own log, even when their lines do not start with a date.

[okkez]

I was wondering if the grok plugin had a similar parameter to format_firstline.

format_firstline is not a parameter of this plugin. It's in_tail plugin or parser_multiline plugin's parameter.

Do you want to combine stack traces? If so, you can use https://github.com/fluent-plugins-nursery/fluent-plugin-concat.

[okkez]

No response. Closing.