search

SumoLogic / logstash-output-sumologic

Forward logstash data to Sumo Logic
Apache License 2.0
18 stars 15 forks source link

Support for dynamic Sumo metadata fields #33

Closed jalaziz closed 3 years ago

jalaziz commented 5 years ago

This is related to #30.

We have a couple use cases where having the source category and source host dynamically pulled from the logs would make things much easier.

For example, since logstash aggregates logs from a number of hosts, the _sourceHost metadata field in Sumo Logic is incorrect.

We also have a scenario where we use Kafka topic pattern matching for input and want to map the topic name to the source category. Due to the dynamic nature of the names, it's impractical to create an output configuration for each topic name.

I was recently thinking about this and I think it can be implemented with relatively little overhead. A simple approach could be to use a map with a key that represents a set of metadata values. The major downside would be having to make an HTTP request per set. However, it seems that the Lambda collector supports exactly that.

I do realize there are a couple different ways we can work around the problem, but solving it at the logstash plugin level would be the most versatile approach it seems.

bin3377 commented 5 years ago

Agree grouping by different headers will be a reasonable approach. Will be similar as the implementation in fluentd buffer plugin

andrzej-stencel commented 3 years ago

Hopefully your issue is now resolved with #36. Just published v1.3.1, please check it out and open a new issue if there are any problems.