perf: scan files at first when source start - Githubissues

loggie-io / loggie

A lightweight, cloud-native data transfer agent and aggregator

https://loggie-io.github.io/docs-en/

Apache License 2.0

1.24k stars 160 forks source link

perf: scan files at first when source start #657

Closed YenchangChan closed 6 months ago

YenchangChan commented 8 months ago

Proposed Changes: Scan new files when a file source start.

如果一个pod拥有极短的生命周期，比如2~3秒就结束，而watcher的扫描定时器是10秒，此时是不能采集到任何内容的。先扫描一次并触发inotify事件可有效避免此种情况的发生
如果一个pipeline匹配到了海量的小文件（如多达数十万甚至上百万个文件），那么需要等待非常久的时间才能采集得到数据，而source启动后立即扫描一遍，可加速采集到数据。

Which issue(s) this PR fixes:

Fixes #

Additional documentation: