Kinesis agent errors out when monitoring a directory with multiple active log files

chamindg commented 6 years ago

Hello,

We have a logstash pipeline writing log files per host, per day into a directory. Logstash pipeline's output portion looks like this;

output {
  file {
    path => "/var/logstash/%{+YYYY-MM-dd}_%{host}.log"
    codec => "line"
  }

So the directory has log files like this, and all of them are active, and get written to very frequently;

-rw-r--r-- 1 logstash logstash   170255 Aug 28 16:26 2017-08-28_10.10.40.17.log
-rw-r--r-- 1 logstash logstash   148640 Aug 28 16:32 2017-08-28_10.10.40.30.log
-rw-r--r-- 1 logstash logstash    62633 Aug 28 16:47 2017-08-28_10.10.42.206.log
-rw-r--r-- 1 logstash logstash    34193 Aug 28 17:41 2017-08-28_10.10.40.163.log
-rw-r--r-- 1 logstash logstash   214425 Aug 28 17:51 2017-08-28_10.10.42.205.log
-rw-r--r-- 1 logstash logstash   247154 Aug 28 17:52 2017-08-28_10.10.42.231.log

And there is a kinesis agent monitoring this directory and sending the logs to a kinesis firehose. Kinesis agent configuration;

{
  "awsAccessKeyId": "xxx",
  "awsSecretAccessKey": "yyy",
  "flows": [
    {
      "filePattern": "/var/logstash/*",
      "deliveryStream": "some-stream"
    }
  ]
}

Problem is, very frequently kinesis agent runs into this error;

2017-08-28 17:43:55.277+0000 sse1-logstash2.pydt.lan (FileTailer[fh:dwh-logging-staging:/var/logstash/*]) com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker [ERROR] E
rror while tracking file rotation for /var/logstash/*: file relative position changed (expected to be after position 2): TrackedFile(id=(dev=fd00,ino=134264416), path=/var/logs
tash/2017-08-28_10.10.40.163.log, lastModifiedTime=1503942105192, size=34193)
        Current file: TrackedFile(id=(dev=fd00,ino=134264408), path=/var/logstash/2017-08-28_10.10.42.231.log, lastModifiedTime=1503942108466, size=239109)
        Current file snapshot (newest first):

Then I modified the logstash configuration to write to a single log file per day, instead of per-host log files. This reduces the number of active log files to 1. Now the error doesn't happen. So it looks like the kinesis agent cannot handle monitoring multiple active log files in the same directory, using the filePattern "/directory/*".

Thanks Chaminda