search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.09k stars 1.59k forks source link

Allow mapping logs to metrics in VRL #12792

Open sushovan23 opened 2 years ago

sushovan23 commented 2 years ago

Discussed throughly in the discussion - https://github.com/vectordotdev/vector/discussions/12773

Opening a ticket since solution involves using lua which is much slower than remap

jszwedko commented 2 years ago

Thanks @sushovan23 .

I think what is needed here is to be able to convert logs to metrics in VRL. I'll update this issue title to reflect that. Aside from that, iteration in VRL should support the dynamic field mapping.

JeanMertz commented 2 years ago

VRL itself has no concept of "logs" or "metrics", it's Vector that has control over how field queries/mutation work, and which ones are allowed/rejected. Unfortunately, this difference between logs/metrics is determined before the program is resolved, and cannot be changed during the run of a program due to how all of this is tied together.

I don't think we want to add the level of complexity needed to allow converting from/to both formats within the program.

It's unclear to me why it wouldn't be possible to format the event as de-facto metrics in VRL, and then add a log_to_metric transform, to actually change the type from log to metric in Vector itself?

spencergilbert commented 2 years ago

It's unclear to me why it wouldn't be possible to format the event as de-facto metrics in VRL, and then add a log_to_metric transform, to actually change the type from log to metric in Vector itself?

I think that's a reasonable solution, but the usage of log_to_metric feels more than a little cumbersome for that. If we could make it easier or enforce via schema "this has all the bits required for metric data model, please change it without any extra config" that would be nice.

JeanMertz commented 2 years ago

"this has all the bits required for metric data model, please change it without any extra config" that would be nice.

Yeah, that's definitely possible, although not without a chunk of work to allow such automated conversions to happen.

Another alternative (but not one I'm particularly on board with) is to allow configuring remap to say "after mapping this event, force it to become a metric", which then also implies you'd have to ensure a certain schema is met before the VRL program would compile.

spencergilbert commented 2 years ago

Another alternative (but not one I'm particularly on board with) is to allow configuring remap to say "after mapping this event, force it to become a metric", which then also implies you'd have to ensure a certain schema is met before the VRL program would compile.

Perhaps that could be the/a improvement for the "conversion" transforms where rather than the current "config based" approach it takes VRL and then does the "if it meets the schema it's converted"