search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

Log Courier and Log file Rotation #336

Closed Inderjeet26 closed 8 years ago

Inderjeet26 commented 8 years ago

Hello,

We have Log Courier deployed on many edge node machines. The problem we are facing is that that LC still hold on to the rotated files even after that file is harvested and has not changed form more than 1 hour. This results in the problem. The file that LC holds is not able to get deleted from the file system.

We are using LC version 1.8 at the moment. This version of LC supports "dead time" parameter which we have setup to 1 hr to resolve this issue. By default it is "24 hr". Dead time config : https://github.com/driskell/log-courier/blob/v1.8/docs/Configuration.md#dead-time.

Here is more information: Log-courier.conf "files": [ /* * BW logs / { "paths": [ "/opt/tibco/tra/domain//application/logs/.log" ], "dead time":"1h", "fields": { "type": "bw" },

Lsof output log-couri 8254 root 10r REG 253,2 20467190 528770 /opt/tibco/tra/domain /HUBQA02_A/application/logs/FinLogCacheLoaderFromDB_OutgoingRemittanceInstrMN-Fi nLogCacheLoaderFromDbPAR-1.log.4 (deleted)

log file [root@host log-courier]# ll /opt/tibco/tra/domain/HUBQA02_A/application/logs/FinLogCacheLoaderFromDB_OutgoingRemittanceInstrMN-FinLogCacheLoaderFromDbPAR-1.log.4 -rw-r--r-- 1 20467178 Aug 1 13:23 /opt/tibco/tra/domain/HUBQA02_A/application/logs/FinLogCacheLoaderFromDB_OutgoingRemittanceInstrMN-FinLogCacheLoaderFromDbPAR-1.log.4

Here is the LC we are using: https://github.com/driskell/log-courier/blob/v1.8/docs/Configuration.md .

Can you pls tell a quick fix if possible in the above case.

Thanks

Inderjeet26 commented 8 years ago

Any idea on this issue?

driskell commented 8 years ago

Files are closed when they reach EOF and the dead time is reached. So the usual explanation is that the logs are not shipped yet. Log Courier doesn't want to lose the logs so keeps file open until it is fully sent. There's not yet an option to force close the file even if not fully sent.

driskell commented 8 years ago

Usually cause of the logs not shipping is a broken connection or Logstash can't keep up with the rate of logs being shipped. You'll need to examine the pipeline and make sure it is fast enough to keep up.

I do plan to add the force close option though so logs can be completely dropped if pipeline blocks or is too slow, rather than just holding the files open and allowing the disk to fill up which is disastrous.

Inderjeet26 commented 8 years ago

Hello, What happens when the Logstash comes back up and the Log Courier which is still holding onto the log files. Does the Log Courier send those old logs now in the already rotated opened log files?

driskell commented 8 years ago

@Inderjeet26 Please open a new issue

Inderjeet26 commented 8 years ago

Submitted: https://github.com/driskell/log-courier/issues/343