vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.57k stars 1.54k forks source link

Ability to control flow from programmable transforms #1942

Open binarylogic opened 4 years ago

binarylogic commented 4 years ago

Something that would be very powerful would be the ability to dynamically define channels from programmable transforms. This would unblock complex control flow logic. For example:

[transforms.control_flow]
  type = "lua"
  source = """
if ... then
  emit(event, 'lane1')
else
  emit(event, 'lane2')
end

This would produce named outputs, similar to the swimlanes transform that could then be connected downstream:

[sinks.first]
  inputs = ["control_flow.lane1"]

[sinks.second]
  inputs = ["control_flow.lane2"]
binarylogic commented 4 years ago

@a-rodin I'm unassigning you and deferring this issue for later. While I think this is cool feature users can use the new swimlanes transforms to achieve this. And if the conditions in that transform are not enough, a workaround is to add a field as a flag and use that in a downstream swimlanes transform.