search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

Index after a log rotates #233

Closed altoic closed 8 years ago

altoic commented 9 years ago

I have this random use case which probably is not very popular. I pause log-courier at times for 30-40 minutes because I need to do some updates to my logstash configs. So what happens when it was progressing through a log. I stop for 40 minutes and in that time the log has rotated? What will happen start at the same position? Start at the beginning? Just curious.

THanks, M

driskell commented 9 years ago

Log Courier keeps the file open until it finishes reading and then the dead time has passed. Then it closes the file, releasing it for deletion if it was already deleted.

So in your case Log Courier will keep the rotated file open, pausing while it waits for the server to come back up (the internal queues will be full). It'll see the new file and open it and start reading it but pausing on the first line to wait as well.

Once server comes back up it then resumes where it left of.

Even if old file deleted during rotation Log Courier keeps a reference so it can still read it. Though gone from file system it'll still exist until it is closed by all applications including Log Courier. On Windows it is similar but the file, one deleted, will still exist on the file system it'll just report "Access denied" on any attempt to open it because it is "Pending deletion"

Hope this explains it sufficiently!

driskell commented 9 years ago

Did the above answer you query? Do you need any more help?