Handle multiline exceptions

[mrdavidlaing]

In files exception messages are logged over multiple lines; and are thus imported as multiple log events, when all the lines should be combined into a single log event.

Logstash has a multiline filter which we should look at implementing to address this.

[mrdavidlaing]

This dashboard shows the current multiline processing "error" - http://logsearch.cil.stack.me/#dashboard/elasticsearch/Issue78-ExceptionParsing

[dpb587]

This needs to happen on the log shipper side of things (currently your workstation, currently via logstash config) since, as mentioned before, messages may be processed out of order. You mentioned the multiline filter and that would be my recommendation as well. Assuming subsequent lines are indented, something like the following should work:

filter {
  multiline {
    type => "ci_log4net"
    pattern => "^\s"
    what => "previous"
  }
}

[mrdavidlaing]

Just added the multiline filter - https://github.com/cityindex/logsearch-on-aws/commit/3929b5759062fe46286cee9b6188172928f3add5

Observations

1. it seems to have doubled the shipper service memory & CPU usage, to 15% & 800MB
2. The number of log events seems to have reduced 10x (weird!)
3. I haven't yet found an exception in the new data to see if it is working :)

[sopel]

Wouldn't the reduced log event number simply indicate that the multiline filter already applies somewhere? If the targeted exceptions are not the cause, maybe some log events are indented w/o reason already, thus a more specific regular expression would be required?

[sopel]

At least it seems to work for exceptions now, very nice :)

[mrdavidlaing]

Lets consider this resolved.