search

logstash-plugins / logstash-output-file

Apache License 2.0
23 stars 53 forks source link

Overwrite file #24

Closed runningman84 closed 6 years ago

runningman84 commented 8 years ago

I would like to use the file output to overwrite an existing file. Right now the new content is appended at the end of the existing file which keeps this file growing.

https://github.com/elastic/logstash/issues/4337

magnusbaeck commented 8 years ago

How would this behave if Logstash is unexpectedly restarted, i.e. how would Logstash know whether the file really should be overwritten?

runningman84 commented 8 years ago

Just write a new file instead of appending. Use case would be to monitor when the file was modified in order to know if logstash is up and running.

magnusbaeck commented 8 years ago

If all you want is an indicator of whether Logstash is running there are other ways of doing that. While the feature you're asking for sounds reasonable I don't think your particular case is a very good justification.

lordnynex commented 8 years ago

FWIW I'd like to use this feature to aggregate metrics. This would prevent me having to add a post processing step to SUM metric counts.

ambivalentno commented 8 years ago

+1 for this feature. I mean, you can check if logstash is up in different way, but I feel this quite a convenient way to work with heartbeats. at the moment i push them to a rabbitmq and worker to rewrite that file.

rababarber commented 8 years ago

+1 for this feature. For exactly the same reason as described by @ambivalentno - implementing simple healthcheck

webmstr commented 7 years ago

Happy anniversary, issue #24. You could also use the functionality to generate yml files for the translate{} filter - within logstash itself!

For my project, I write the file to "dict.yml.new" every 5 minutes, and have a cron job that mv's that to "dict.yml". That protects me if the original data source is unavailable, as logstash caches the old file until a new one appears.

jordansissel commented 7 years ago

While I do not recommend using this strategy for health checking (Logstash 5.0 has an API for monitoring behavior), I don't have any disagreements with the proposed feature.

I have submitted PR #50 to implement this.

wols commented 6 years ago

Is this a hidden feature? Not documented here https://github.com/logstash-plugins/logstash-output-file/blob/master/docs/index.asciidoc or https://www.elastic.co/guide/en/logstash/6.0/plugins-outputs-file.html