vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.13k stars 1.6k forks source link

Allow file sink to accept metric events #21704

Open rahulsmehta opened 1 week ago

rahulsmehta commented 1 week ago

A note for the community

Use Cases

I am attempting to save metrics to disk using the native_json codec using http_server source & the file sink, but it appears to only support logs - see below https://github.com/vectordotdev/vector/blob/a8e82dd71a38065d32db872e489e5dc4edb6dbc3/src/sinks/file/mod.rs#L197

Attempted Solutions

Similar to the workaround mentioned in https://github.com/vectordotdev/vector/issues/20756, I was able to work around this by using the metric_to_log transform to save metrics to disk, and then applied the inverse log_to_metric transform downstream.

Proposal

Allow the file sink to accept metric events

References

Version

vector 0.42.0 (aarch64-unknown-linux-gnu 3d16e34 2024-10-21 14:10:14.375255220)

rahulsmehta commented 1 week ago

cc @nionata

pront commented 1 week ago

Hi @rahulsmehta, thank you for the request.

It's been a while since I looked at the file sink implementation but I don't think it will require too much modification to support metrics. If you wanted to test how it works with metrics, you can remove the & DataType::Log and run a config. The native JSON codecs support all data types.