vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.66k stars 1.56k forks source link

Consider separate `source` options for different event types in programmable transforms #5891

Open binarylogic opened 3 years ago

binarylogic commented 3 years ago

Currently, the lua and remap transforms operate on logs and metrics, but I can't fathom a use case where a single script would operate on both event types without an if statement checking for the event type. This is fine until we start to consider compile-time type checking, where we must know what type of event we're working with. I propose two ways of handling this:

  1. Do not allow mixed input types for the remap transform. All inputs must either be of the log or metric type.
  2. Add separate source options, such as source.logs and source.metrics.

I prefer 1 as it is the cleanest and most obvious. It is also backward compatible with the lua transform if we decide to enforce that there for consistency.

cc @JeanMertz @FungusHumungus for thoughts on the remap transform.

StephenWakely commented 3 years ago

We were just talking about this in regards to #5887.

There are some differences between the way log and metrics behave with Remap. At the minute we can't guarantee what type we are handling in the Transform at boot time to perform these extra checks. A way of limiting this would be quite useful.

It is possible for the Lua transform to take in, say a log event and then conditionally output a metric or a log. Noone would want to do this (probably), but it's a complication we'd need to consider.