Make "File" sink reopen files on SIGHUP

timp87 commented 2 years ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We could take advantage of external typical log rotation tools like logrotate(8).

Attempted Solutions

Vector's "File" sink provides log file name templating described here I used path pattern like "/path/to/log/dnstap.log-%Y%m%d". This is an incomplete solution since vector doesn't provide log file retention options. So you have to deal with old file deletion. Also there is copytruncate option for logrotate, but in this case some log events may be lost.

Proposal

Make "File" sink able to reopen (i. e. flush, close and open) log files on SIGHUP. This gives much more control over the log file rotation process. And IMO this approach is being considered typical in linux world.

References

No response

Version

0.21.2

jszwedko commented 2 years ago

Thanks @timp87 . I think this is a good idea.

cc/ @lukesteensen since this relates to Vector reload, which he's been looking at recently.

vaskozl commented 1 year ago

Currently the only solution (and somewhat recommended) when using logrotate is to use "copytruncate" which can lead to some log loss.

At present moving the file results into logs being written to the same moved file while deleting (say because you compressed it) leads to log loss.

This makes it incompatible with logrotate "create" mode. Adding support for SIGHUP would make it compatible with with the mode when using delaycompress.

timp87 commented 1 year ago

not sure where it is recommended. I'd say it's just the only "dirty" workaround for now. Logrotate was an example of an external tools that can help here. Moreover copytruncate looses data.

vaskozl commented 1 year ago

Agreed, I was refering to #4122

My current solution is to use date templating instead of logrotate and find to gzip/prune, while symlinking todays date to a non-dated file for convenience.

While we can write staight to gzip, the funcionaly also has usability and safety caveats.

vectordotdev / vector