tenzir / public-roadmap

The public roadmap of Tenzir
https://docs.tenzir.com/roadmap
4 stars 0 forks source link

Watching Directory Loader #51

Open mavam opened 1 year ago

mavam commented 1 year ago

In order to make it easy to continuously import data from files in a given directory, we would like to make it possible to "tail -f" all files in it. The desired UX would be along the lines of:

from file --tail /path/to/dir read json | ...

We can employ fswatch to implement this feature.

💯 Definition of Done

- [ ] Define the operator UX
- [ ] Implement it in the `file` connector
- [ ] Test with Zeek's spool directory (by default `$PREFIX/logs/current`)
- [ ] Test with a Suricata EVE log