Closed Hexta closed 1 year ago
Hi @Hexta !
This is a fundamental deficiency of this checkpoint strategy vs. the fingerprint
strategy. I'm not really sure how we would work around it in Vector. Do you have ideas of how this might be tracked in a way to detect inode reuse?
Hi!
From the top of my head.
fanotify
could be used for subscribing to FS events and receive file_handle
structs, which could be passed to open_by_handle_at()
function for opening the original file. open_by_handle_at()
will fail If the file was removed and a new file created with the same inode between receiving file_handle
and calling the function.
Another benefit — we don't need to continuously scan for new/removed files. Just poll a file descriptor returned by fanotify_init
. So it's good for scalability when there are a lot of input files.
But I'm not sure if it's worth implementing )
IMO it'd be great:
device_and_inode
strategy to fingerprint
section of file
source documentation.Thanks for the thoughts @Hexta ! We don't currently leverage fanotify
anywhere but I think what you suggested seems plausible. If we do decide to add fanotify
we'll probably rethink much of the current implementation which currently relies only on scanning.
Agreed with adding notes to the docs about this. I'll do that.
Actually, I see this is already documented here: https://vector.dev/docs/reference/configuration/sources/file/#fingerprinting
Given that, I'll close this out, but thank you for the discussion!
A note for the community
Problem
device_and_inode
fingerprint strategy doesn't handle inodes reusing during log files rotation. It leads to logs losing.Example:
Vector resumes watching
/var/log/foo.log
from the last checkpointed position.Configuration
No response
Version
0.33.0-custom-61c0ae8a5
Debug Output
No response
Example Data
No response
Additional Context
No response
References
2163