search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

Support for dynamic file path #225

Closed angel-smile closed 8 years ago

angel-smile commented 9 years ago

Is there a way to specify dynamic file path/name in log-courier configuration file? The targeted log file rotates everyday with its name set as the date. It seems in logstash I can specify the file name in the configuration file as "/full_path/%{YEAR}%{MONTHNUM}%{MONTHDAY}". Is there a way to do this in log-courier?

Thanks in advance!

driskell commented 9 years ago

Good idea. Have you seen https://github.com/driskell/log-courier/issues/37 ?

My only concern is timing though and if there could be missed logs because courier didn't see a dead file update at 23:59 on 2015-08-20 because it's next scan was 00:00 on 2015-08-21 and it scanned a different folder.

It all comes into timing with dead time I guess. Ensuring files kept open long enough to notice changes right until past the rotation.

37 seems the root cause and this is a good partial solution. Be good to hear your thoughts on the dead time action mentioned there though.

angel-smile commented 9 years ago

Hi Jason,

Many thanks for the response. I agree with you that the dead time action should be a solution for our issue. We are running an old version of log-courier. I'll upgrade it to the latest version and try dead time. It would still be good to be able to specify file path with regex like "%{YEAR}%{MONTHNUM}%{MONTHDAY}" though.

angel-smile commented 9 years ago

Hi Jason,

"dead time" has fixed our problem. Do you want me to close the case or leave it open till regex feature for file path is implemented?

Thanks a lot for the help.