open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.11k stars 2.39k forks source link

[pkg/ottl] Converter: MarshalKeyValue #35334

Closed kuiperda closed 1 month ago

kuiperda commented 2 months ago

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

Problem: Sending Windows Event Logs to QRadar

The recommended approach is to turn the event into key value pairs to put into the syslog rfc3164 message that QRadar expects and will automatically parse.

The current solution for this involves a long list of set and concat funcs to manipulate the parsed record into Key Value format.

Describe the solution you'd like

A simple MarshalKeyValue OTTL converter would take a pcommon.Map and convert it to a string of key value pairs with format k=v k2=v2...

It would be an inverse to the existing ParseKeyValue func. For detailed examples of behavior, see ParseKeyValue tests.

Note: There is some loss in the Parse func, i.e. trimming whitespace, removing wrapping ' and ", and escaping ". Marshal would persist this loss so composing the funcs would not be an exact inversion.

I would advocate for customizable delimiters with default being = and (space).

Marshal would handle collisions with delimiters in the keys and values by wrapping the k/v in " after escaping any unescaped "

For nested maps and arrays, this simple case could automatically flatten them to the top level. But we may need to decide on a representation for nested maps and arrays and potentially allow customization of those symbols and delimiters as well. The other key consideration here is that Parse does not expect its input to be nested.

Example

MarshalKeyValue(target, Optional[delimiter], Optional[pair_delimiter])

Simple example usage (Initially added as a processor in the Bindplane Agent)

Describe alternatives you've considered

Adding a processor that handles Key Value marshaling. An initial approach to this has already been added in the Bindplane Agent.

Additional context

No response

github-actions[bot] commented 2 months ago

Pinging code owners:

djaglowski commented 2 months ago

I think key/value format is simple enough that providing a marshaler makes sense. Handling of nested elements can just be a documented caveat that they are rendered as strings. Users can always use the dedicated Flatten function before marshaling if they like that output better.

odubajDT commented 2 months ago

Hello, I would like to look at this issue!

kuiperda commented 2 months ago

@odubajDT I am working on an implementation for this already. This issue is intended to gain consensus around a standard.

kuiperda commented 2 months ago

Initial implementation added in #35409

crobert-1 commented 2 months ago

Removing needs triage based on response from project maintainer.

kuiperda commented 1 month ago

Completed with #35409