search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

Multiline and log rotation #227

Closed altoic closed 9 years ago

altoic commented 9 years ago

There was a requirement where I need to multiline logs but there is also rotation that will cut off these messages. My pattern will match even if they are cut but if I want the complete message comprised of these single logs it is recommended to run multiline on logstash.

"Log courier applies multiline only to entries within the same file"

Only difference is that more messages will be complete! Maybe make this an improvement to courier? Either way there is a work around. Thanks, M

driskell commented 9 years ago

This is because multiline codec only applies to a single file like you say. So if the lines overflow to another line it does not pick them up. I will get this added to the documentation if its not already.

It's difficult because the codec runs at the file level - it has no knowledge of other "streams" or new log files since there's no guarantee the writes are finished to the last log file when the new log file appears and starts receiving logs. Generally it's best here to ensure the application writes multiline entries in a single write so it all hits the same file in one write. If that's not possible, best way is to try the Logstash multiline filter since it receives logs from all files (the courier input on the Logstash side doesn't support codecs yet).

driskell commented 9 years ago

It could be an improvement to courier but I feel at this time it's complexity outweighs any benefit as most applications do log in chunks to the same file rather than splitting across files.