Open jszwedko opened 3 years ago
I'm in need of this as well. It seems that the behavior of the source used for named pipes would be similar to the file descriptor source, except that:
I think this is sufficiently different from both the file and the file descriptor source, so a new "Named Pipe" source might be best.
Would a PR with this have a chance to be merged? Other ideas or suggestions?
I bumped into this when using the file source to ingest Nomad logs. Nomad creates named pipes, which Vector tries to seek.
This blows up at runtime with:
thread 'vector-worker' panicked at lib/file-source/src/file_watcher/mod.rs:124:67:
called `Result::unwrap()` on an `Err` value: Os { code: 29, kind: NotSeekable, message: "Illegal seek" }
I worked around it using the exclude
option.
Current Vector Version
Use-cases
(Copied from discord: https://discordapp.com/channels/742820443487993987/746070591097798688/755150918969982986)
The user has a setup where logs are written to a dynamic set of named pipes which they would like to ingest with vector.
Attempted Solutions
We originally thought that the
file
source might be able to handle this, but it appears not, due to the checkpointing requiring seeks:The error is output when the pipe is written to. Additionally, vector hangs when trying to shut down.
The config:
Vector is able to use it as a
stdin
source, but this will only work for one pipe pervector
instance.Proposal
Not sure! We could extend the
file
source, thestdin
source, or add a new one to handle this depending on feasibility.