Logstash doesn't ship tail of my log file until I terminate logstash process

[jordansissel]

(This issue was originally filed by @fandrei at https://github.com/elastic/logstash/issues/2638)

---

My config:

input {
  file {
    path => "....../logs/*.in"
    start_position =>  "beginning"
    sincedb_path => "....../logs/sincedb"
    discover_interval => 1
  }
}

output {
  file {
    flush_interval => 1
    path => "......./logs/log.out"
  }
}

I write a bunch of lines to an input file; the most of these lines are shipped to the output successfully, however a few tailing lines aren't shipped for indefinite time (at least 5 minutes); they only get shipped when I stop the logstash process. I'm using logstash 1.4.2 on Windows 7. Any ideas why this can happen?

[KarlikB]

Possible workaround is set flush_interval => 0 to force file flushing for each log message (at the some performance cost)

[ghost]

Having the same problem here with 1.4.2 also. The last few events were not pushed to file until we triggered a new event setting the flush_interval to 0 did indeed 'solve' it, but we'll have to see if performance isn't hit too bad.

By the way, any1 an idea if this is fixed in 1.5?

[KarlikB]

Sorry, no idea. I have to use stable version released at the beginning of my project. (fixed version set defined by cutomer)

[fbaligand]

+1 for this issue.

I think this problem could be resolved using plugin "flush" method.

[hyperdash]

I have the same problem and read some source code to find that the plugin seems to have no thread to measure the interval. Flushing and checking the interval are called only with events. Without events, flush method is never called.

[fbaligand]

Flush method is called every 5s by logstash-core itself. It works fine for my plugin : aggregate-filter.

That said, I agree that 5s implies that flush_interval must be a multiple of 5.

[hyperdash]

Okay. I understand now what you mean. I meant "private" flush method inside output-file plugin.

[chananelp]

+1 for this issue. I'm running logstash-2.3.1 and it still does not flush some tail lines of my log..

[ThomasLau]

hi, has anyone solved this? i meet the same question.

[yinrong]

+1 logstash latest version still not tailing my files

[yinrong]

+1 logstash latest version still not tailing my files, unless:

sincedb => /dev/null
start_from => end

[fbaligand]

This issue should be fixed since Logstash 6.3, that includes file output plugin v4.2.5, that fixes this particular issue.