[log-shipper] Json parsing, plus filter by parsed fields content

[duckhawk]

Preflight Checklist

• [X] I agree to follow the Code of Conduct that this project adheres to.
• [X] I have searched the issue tracker for an issue that matches the one I want to file, without success.

Use case. Why is this important?

There is no parsing of json in pod messages, and no possibility to search by field headers/content in logs in storage. Plus, in some cases we needs to drop messages by json fields headers/content (for example, to save space in case of huge amount of logs). Now it's possible to setup a logstash between log-shipper and elastic, but it's not looking like native solution.

Proposed Solution

In addition to extraLabels, add possibility to parse all json in some fields, and add possibility to filter via this parsed json fields header/content.

Additional Information

No response

[nabokihms]

The right way to resolve this issue is to add an option to parse messages into labels.

Note: Do not forget to add depth options because messages can have many layers, and most of the time, you do not want to parse 8+ staged JSON docs.

Example of expected config section:

elasticsearch:
  parseMessage:
    dedot: | true, false (true by default)
    depth: | 1, 2, All (0 by default)

[nabokihms]

There might be a problem with the depth option because VRL has no functions to parse JSON docs partially.

[nabokihms]

The 0.22 Vector release blocks this issue (I believe that this release will include the max_depth function).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.