Doesn't follow log after log rotation

[mheffner]

When remote_syslog2 is following the log files output by a Dropwizard application, it fails to continue following the main log file after it is rotating by Dropwizard's log rotation schedule. Restarting remote_syslog2 will fix the problem and will publish new log files.

cc @benodom

[johlym]

Hi @mheffner how does Dropwizard rotate logs?

[mheffner]

@johlym So I believe it renames to a backup file with date extension, then creates a new file. For example:

-rw-r--r-- 1 metrics metrics   215330 Jun 12 00:14 boatwrite-2016-06-11.log.gz
-rw-r--r-- 1 metrics metrics   243207 Jun 13 00:10 boatwrite-2016-06-12.log.gz
-rw-r--r-- 1 metrics metrics   395382 Jun 14 00:08 boatwrite-2016-06-13.log.gz
-rw-r--r-- 1 metrics metrics   191182 Jun 15 00:10 boatwrite-2016-06-14.log.gz
-rw-r--r-- 1 metrics metrics   155578 Jun 16 00:10 boatwrite-2016-06-15.log.gz
-rw-r--r-- 1 metrics metrics 13195775 Jun 16 16:33 boatwrite.log

In this scenario we are trying to watch boatwrite.log.

[johlym]

@mheffner Ahh ok. That makes sense and explains the behavior. remote_syslog2 relies on the filesystem giving it the inode to watch. During this log rotation, the filesystem doesn't tell remote_syslog2 about the new inode to watch (from the creates a new file step), logs will stop flowing.

I see two things to try: 1) rotate the log files in a copy-truncate fashion so the file being actively written to never changes 2) run remote_syslog2 with the --poll flag. This will tell it to do its own file monitoring versus relying on the filesystem.

Thoughts?

[pcn]

@johlym I can't find a copytruncate-like option for logback. I guess we could use --poll, but it seems like given how much cleaner and less resource-intensive it is to rename() a file, then open() a new one in logging code (vs copying a file and then truncating an open file), could remote_syslog2 use inotify to watch the set of directories where logs are created and react to file changes on those directories instead?

I guess the issue is really here: https://github.com/hpcloud/tail/issues/90 edit or maybe here? https://github.com/hpcloud/tail/issues/21

[johlym]

@pcn Definitely. I talked with @eric about this a bit ago and he pointed me to tail's ability to follow by name.

[snorecone]

Closing this since I've confirmed that this is working on the latest version: v0.19

Both logback-style and copytruncate-style are supported.