search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

disadvantage of short dead time setting #355

Closed pahool closed 3 years ago

pahool commented 7 years ago

We have encountered issues with log-courier holding onto deleted files for too long and having this create disk space issues for us. We are planning to decrease the dead time setting in order to have log-courier release deleted files more quickly (we're thinking of trying a 5m setting.)

Is there any disadvantage to a low dead time setting? Is it just the overhead of re-opening files?

driskell commented 7 years ago

Hi @pahool

If the remote Logstash instance isn't able to process the events being sent to it, Log Courier will hold the file indefinitely until all data is flushed. Only then will the dead time begin. You are right in that, with a fully working pipeline, a low dead time is fine with only the overhead of re-opening files. Of course, on busy servers though, files are modifying all the time so it's highly unlikely to close the file until the end of the day. Just note it won't close the file if the contents aren't sent due to pipeline problems.

It's been on my list of things to look at for a long time but I've always considered it a problem with the central Logstash instance not being able to process the backlogs, and personally I've never wanted to "lose" logs because of a backlog, and actually Log Courier was designed specifically around that requirement (as the previous iterations of shipper would always lose entries when backlogs occurred.)

Latest Log Courier does have tracking internally of "current size" vs "processed size" which even updates independently of progress through a file. This was added in preparation for settings to allow losing logs if a file gets too big or too behind, maybe even taking into account available space on the disk, so that space can be freed. Also there was preparation for allowing a log to be closed X amount of time after it was deleted, so that deleting the log file on disk will force Log Courier to lose the logs after a certain amount of time, which I guess is the most ideal way.

Happy to take in a request to implement some of this but I don't think I'll get to it for at least a few weeks - I'm mostly in maintenance mode on Log Courier and I've been tinkering away on the Logstash side for a while and hoping to update the plugin soon.