Fluentd cannot detected log being rotating

[jhowliu]

As title, Is it possible integrate with fluentd in-tail?

Fluentd stop updating the pos-file when rotation happened.

[Mathiasdm]

I see the same issue with simply 'tail'. If I run 'tail -F logfile' and the file gets rotated, tail doesn't detect the change (even though the '-F' flag should guarantee it).

[natefinch]

I'm not sure how tail and fluentd work in this case. I can look into it. There may be some file descriptor magic logrotate etc do to keep tail on the right track.

[Mathiasdm]

To add to this, I was able to solve the issue for tail by using 'tail -F --max-unchanged-stats=5 logfile', which reloads the file when it no longer changes. Using only '-F', I noticed that tail kept looking at the moved file (according to what I saw in /proc//fd/ ).

[natefinch]

Yeah, that makes sense. Tail would follow the file descriptor, even if the name changed. When lumberjack rotates, it just renames the old file, which would keep the same file descriptor. I would need to look at how to fix that. I could copy the file and truncate the original, but then we'd have that file's contents on disk twice, and that could mean you'd be using too much disk space (although I guess that's the same disk space you'd use once the original file fills up to capacity, too. Hmm.)

[phuslu]

Yes yes, that's what & why I wrote in https://github.com/natefinch/lumberjack/issues/115#issuecomment-739835573

[phuslu]

I could copy the file and truncate the original

the copyturncate option which like logrotate do is a bad idea,

1. it's not safe.
2. it will introduce a heavy load and causes a large performance downgrade in a high-qps server.

Many companies (include us) were encountered the 2nd issue.

[phuslu]

A related issue https://github.com/logrotate/logrotate/issues/269 and https://github.com/logrotate/logrotate/pull/361