observIQ / stanza

Fast and lightweight log transport and processing.
Apache License 2.0
182 stars 23 forks source link

Add and Remove operators should support an array of things to add or remove #469

Open Dylan-M opened 3 years ago

Dylan-M commented 3 years ago

Is your feature request related to a problem? Please describe. Currently I have to specify multiple removes, with defined IDs, in order to remove 3 fields that I combined into a single field.

Describe the solution you'd like Allow an array to be passed, such as we do with other operators, of fields to add or remove

Describe alternatives you've considered Currently using multiple entries for remove. Each requires an ID in order to be unique.

djaglowski commented 3 years ago

I agree with this idea for remove, but I don't see how exactly it would work with add.

The remove change would be a welcome contribution to opentelemetry-log-collection as well.

To clarify my expectations, the following should be allowed:

type: remove
field: myfield1
type: remove
field: 
  - myfield1
  - myfield2
Dylan-M commented 3 years ago

Yes, I've been thinking about how to format the add for an array, and I could only come up with 2 solutions.

Solution 1:

type: add
field:
  - key
  - value
field:
  - key
  - value

Essentially allowing field to appear multiple times.

Solution 2:

type: add
field:
  - key1
  - key2
  - key3
value:
  - value1
  - value2
  - value3

Such that key1-value1 are paired together, and so on.

djaglowski commented 3 years ago

Unfortunately, I don't think solution 1 is possible due to the nature of yaml and the map structure it represents.

Solution 2 is generally considered an antipattern, as it requires the user to make implicit connections between values. As soon as there are more than a handful of values, it becomes very difficult to understand what is happening.

Dylan-M commented 3 years ago

Unfortunately, I don't think solution 1 is possible due to the nature of yaml and the map structure it represents.

Solution 2 is generally considered an antipattern, as it requires the user to make implicit connections between values. As soon as there are more than a handful of values, it becomes very difficult to understand what is happening.

Yeah, I should have added neither is a good solution. Just the only 2 that come to mind. Which is unfortunate, but it is what it is. Remove really is the important one here anyway. At least for my use case. We used add to combine several fields into a single one with formatting. Then removed all the fields that we had combined.