[Filebeat] - validate files before harvesting

VihasMakwana commented 2 weeks ago

Current Issue

This issue exists is we enable prospector.scanner.symlinks.
If we enable prospector.scanner.symlinks and try to ingest a symlink in symlink -> non-regular file manner, the irregular file will still be ingested.
This shouldn't happen and we should validate the source file as well.

Describe the enhancement:

Currently, we validate a file here and only harvest a regular file (i.e. NOT directories, pipes, sockets, device etc).
I propose to move this step before harvesting it i.e. here.

Describe a specific use case for the enhancement or feature:

With https://github.com/elastic/beats/pull/40121 open, we will soon have status reporter.
Our filestream input will remain degraded unless the user changes fixes the symlink and source file.
In my opinion, we should have never ingested the symlink in the first place as the source file is an irregular file.

elasticmachine commented 2 weeks ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

pierrehilbert commented 2 weeks ago

@rdner / @belimawr could you please have a look here to share your thoughts?

belimawr commented 2 weeks ago

@VihasMakwana, I'm a bit confused here... We check if a file is a regular file right before opening it, which is indeed after resolving the symlink. But every single file we open for reading is will go through the validation at https://github.com/elastic/beats/blob/032a4cfd5f3b8fa8354ac1e0062a0e1f196c60d0/filebeat/input/filestream/input.go#L293-L296 which checks the file mode https://github.com/elastic/beats/blob/032a4cfd5f3b8fa8354ac1e0062a0e1f196c60d0/filebeat/input/filestream/input.go#L323-L329

So even if the fileWatcher finds a file under a symlink that is not a regular file and returns it, filestream.openFile will still validate it by calling checkFileBeforeOpening and won't be harvested if it is not a regular file.

Regarding reporting the input status as degraded in this case, I believe it is the correct behaviour. The user has configured Filestream to ingest something that is not a regular file, thus the user should be notified of their error and the input should stay degraded until this is fixed.

We just need to make sure the message returned to the user is clear enough so the can understand and act on it.

elastic / beats

[Filebeat] - validate files before harvesting #40151