search

SumoLogic / sumologic-kafka-push

Utility for reading logs from a Kafka Topic and sending to Sumo Logic.
Apache License 2.0
4 stars 4 forks source link

Add option to embed json fields metadata inside the log payload #46

Open rishiyemme opened 2 years ago

rishiyemme commented 2 years ago

Adding a boolean option moveFieldsIntoPayload to inject specified fields into the log JSON payload to allow for dynamic auto JSON parsing in Sumo. Enabling this option would also prevent fields from being sent in the HTTP request to Sumo.

rishiyemme commented 2 years ago

Pushed some changes while waiting for some customer feedback on the log format when moveFieldsIntoPayload = true. Per the latest changes, whenever moveFieldsIntoPayload is true, we always format the log payload into a JSON log format looking like

{"timestamp":1664396202215,"log":"{"timestamp":1654016401446,"PID":"3","USER":"root","MEM":"0.0","CPU":"0.0"}","metadata":{"bigintfield":"228930314431312345","boolfield":"true","intfield":"100","stringfield":"stringvalue"}}

essentially ignoring the payloadText field. The metadata field will include all fields.

Otherwise when moveFieldsIntoPayload is false, we revert to the previous behavior.

vsinghal13 commented 2 years ago

Otherwise when moveFieldsIntoPayload is true, we revert to the previous behavior. @rishiyemme you meant when it is set to false, we revert to previous behavior.