search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.62k stars 1.55k forks source link

Add more options to file source patterns, e.g. "follow_links", "case_insensitive", "max_depth" #5938

Open pablosichert opened 3 years ago

pablosichert commented 3 years ago

Current Vector Version

vector 0.12.0

Use-cases

Give more fine-grained control over file include patterns, e.g. if symlinks should be followed and if the pattern should be considered case (in)sensitive.

Proposal

Add options to the file source config that are propagated to the GlobWalkBuilder and use that builder for the include_patterns.

References

mtekel commented 1 year ago

Symlinks are a bit of issue. 0.32.0 sees original file and symlink as two separate and then complains checksum/fingerprint is the same and it spams the log. Interestingly, error messages have repetition supression, but info messages don't.