search

SumoLogic / logstash-output-sumologic

Forward logstash data to Sumo Logic
Apache License 2.0
18 stars 15 forks source link

dynamic source_category, adding batching by count #28

Closed henry-megarry closed 6 years ago

henry-megarry commented 6 years ago

making source_category dynamic per event, adding tests, adding batch_count

henry-megarry commented 6 years ago

Failed because:

Could not find gem 'logstash-devutils (>= 1.3.1, ~> 1.3)' in any of the gem sources listed in your Gemfile.

Not sure if this is an issue that I am causing?

bin3377 commented 6 years ago

The sourceCategory is not supposed to set per event because:

  1. The sourceCategory can only on HTTP header, and all message in the request will share the same value
  2. For above reason, per event source category will be ignored except last event in the pile
henry-megarry commented 6 years ago

The way I had it set up was having a map of piles with the sourceCategory as the key. Then it would send those piles as separate batches. Our use case is that we have several products using the same logstash instance and we want them sorted into different source categories so that we can use role based access with our users.

bin3377 commented 6 years ago

The HTTP source is not designed to accept dynamic source category per log line. Even this can work with HTTP request with only single log line, it's not suggested since 1) it will significant limit the throughputs and 2) the multiline processing will not work (since only the log lines in the same request will be guaranteed on same node after load balancing).

If your events are falling into few categories, it's recommended to use multiple output plugins in parallel with filter and condition expression - https://www.elastic.co/guide/en/logstash/6.0/event-dependent-configuration.html

Using filter to split your messages into different output (which all sumologic but with different source category settings, or you can even create multiple HTTP sources with different URL to handle them individually)