Plugin is closing file with periodic updates

[ceeeekay]

I have heartbeat input generating messages which I'm sending to a file for Icinga to monitor.

Since upgrading Logstash from 5.6.5 to 6.1.2 I'm seeing a lot of open/close messages for this file in the Logstash logs.

Has the timeout on open files changed, and is there any way to override this?

• Version: 4.2.1
• Logstash Version: 6.1.2
• Operating System: Ubuntu 16.04.3
• Config File:

  input {
  heartbeat {
  message => "epoch"
  interval => 15
  type => "heartbeat"
  }
  }
  [...]
  output {
  if [type] == "heartbeat" {
  file {
    path => "/tmp/heartbeat"
    codec => line {
    format =>  [ "%{clock} " ]
  }
  }
  }

• Sample Data: /var/log/logstash/logstash-plain.log

  [2018-01-25T11:23:49,152][INFO ][logstash.outputs.file    ] Closing file /tmp/heartbeat
  [2018-01-25T11:23:53,261][INFO ][logstash.outputs.file    ] Opening file {:path=>"/tmp/heartbeat"}
  [2018-01-25T11:24:19,198][INFO ][logstash.outputs.file    ] Closing file /tmp/heartbeat
  [2018-01-25T11:24:23,271][INFO ][logstash.outputs.file    ] Opening file {:path=>"/tmp/heartbeat"}
  [2018-01-25T11:24:49,568][INFO ][logstash.outputs.file    ] Closing file /tmp/heartbeat
  [2018-01-25T11:24:53,260][INFO ][logstash.outputs.file    ] Opening file {:path=>"/tmp/heartbeat"}
  [2018-01-25T11:25:19,877][INFO ][logstash.outputs.file    ] Closing file /tmp/heartbeat

[ceeeekay]

Any update on this? It's been open for over 6 months.

[ceeeekay]

@jakelandis @guyboertje

Since there's only radio silence on this for nearly seven months, I've done the following as a workaround to keep the incessant open/close garbage out of my log files:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d' { "logger.logstash.outputs.file" : "ERROR" }'

It's in cron so the log level won't revert after a restart.

I really shouldn't have had to do this at all. Looking forward to a permanent resolution to this extremely annoying issue ... please?

[Programie]

I have the same issue with hundreds of files being written to using the logstash-output-file plugin and having a lot of Opening file and Closing file messages in the log.

I guess, opening and closing the same files all the time also increases the CPU/system load as it is in my case. I'm currently not sure whether it's caused by the massive opening/closing of files, but it looks like to be related to it as it only happens on the logstash host that is writing to files.

[ceeeekay]

I wouldn't hold your breath for any sort of response from the devs.. They seem to want to ignore this issue for some reason.

[BulatSaif]

I started using logstash 6.3.2 same problem :(

[imweijh]

https://github.com/logstash-plugins/logstash-output-file/blob/master/lib/logstash/outputs/file.rb#L102 increase @stale_cleanup_interval = 60

better if configurable