Multiline received as last input will be dropped

[cwurm]

When sending data that includes a multiline at the very end, the multiline will be dropped by this codec.

For example, this affects every file being read with the stdin or file inputs (see below for mitigation when using file).

Example

stacktrace.log:

[2017-06-29] exception
  stack trace line 1
  stack trace line 2

logstash.javastacktrace.conf:

input {
  stdin {
    codec => multiline {
      pattern => "^\s"
      what => "previous"
    }
  }
}

filter {}

output { stdout { codec => rubydebug }}

Output:

% logstash-5.4.0/bin/logstash -f logstash.javastacktrace.conf < stacktrace.log
Sending Logstash's logs to /Users/cwurm/Products/Logstash/logstash-5.4.0/logs which is now configured via log4j2.properties
[2017-06-30T14:13:17,934][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-06-30T14:13:23,011][INFO ][logstash.pipeline        ] Pipeline main started
[2017-06-30T14:13:23,116][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-06-30T14:13:26,041][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

Why?

The codec will buffer lines matching a multiline pattern, but not output them until it encounters a line not matching the multiline pattern.

While a flush method is implemented, it will never be called by Logstash: https://github.com/elastic/logstash/issues/6523

Mitigation

Setting auto_flush_interval helps for inputs that don't exit (e.g. file, but not stdin).

However, it introduces a race condition - if the input or Logstash dies before the data is auto flushed it will still be lost.

[PhaedrusTheGreek]

I'm hitting this issue too, using the multiline codec on the stdin input, my final log entry is lost.

[akshay8856]

I am hitting same issue too +1 when multiline codec with stdin input is used.

[fikrigha]

I just ran into this issue using multiline codec on the file input. Same exact thing.

[colinsurprenant]

63 should eventually fix this.