awslabs / amazon-kinesis-agent

Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Other
360 stars 223 forks source link

Logrotate issue ... different than #50 #86

Open diranged opened 7 years ago

diranged commented 7 years ago

We have a similar looking exception - but its not the same - as the one in #50:

Traceback (most recent call last):
File "java.lang.Thread.java" line 745 in run
File "com.google.common.util.concurrent.Callables$3.java" line 95 in run
File "com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.java" line 60 in run
File "com.amazon.kinesis.streaming.agent.tailing.FileTailer.java" line -1 in run
File "com.amazon.kinesis.streaming.agent.tailing.FileTailer.java" line -1 in runOnce
File "com.amazon.kinesis.streaming.agent.tailing.FileTailer.java" line -1 in processRecords
File "com.amazon.kinesis.streaming.agent.tailing.FileTailer.java" line -1 in updateRecordParser
File "com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.java" line -1 in refresh
File "com.amazon.kinesis.streaming.agent.tailing.SourceFile.java" line -1 in listFiles
File "com.amazon.kinesis.streaming.agent.tailing.TrackedFile.java" line -1 in <init>
File "java.nio.file.Files.java" line 2266 in getLastModifiedTime
File "java.nio.file.Files.java" line 1737 in readAttributes
File "sun.nio.fs.LinuxFileSystemProvider.java" line 99 in readAttributes
File "sun.nio.fs.UnixFileSystemProvider.java" line 144 in readAttributes
File "sun.nio.fs.UnixFileAttributeViews$Basic.java" line 55 in readAttributes
File "sun.nio.fs.UnixException.java" line 107 in rethrowAsIOException
File "sun.nio.fs.UnixException.java" line 102 in rethrowAsIOException
File "sun.nio.fs.UnixException.java" line 86 in translateToIOException
NoSuchFileException: /mnt/log/local0.json.11

We see the issue happen ~3-4 times per day across all kinds of different hosts.

image

Our logrotate looks like this:

/mnt/log/*.json {
  create 644 root root
  rotate 10
  size 256M
  nocompress
  missingok
  notifempty
  delaycompress
  sharedscripts
  postrotate
    /usr/sbin/invoke-rc.d syslog-ng reload
  endscript
}
diranged commented 7 years ago

@chaochenq, per our phone conversation, if this is not a real problem, can you patch the code to handle this gracefully and silently?

chaochenq commented 7 years ago

@diranged do you still see this happen after you change to "nocreate"?