Need more detailed information on how to use this plugin

[hinder90]

Presently, I am just ingesting the logs through the HTTP collector with no other options. The log data is in JSON and some perculiar things are happen which I am sure required configuration but the documentation is very scant and I am not sure how to actually configure it so that the Sumo Logic meta data is mapped correctly, each log event is treated as a its own log entry, etc... I need some examples/help/something!

Here is what is going on:

1. All log data is just streamed with no border function between events so logs are all in the same size of text and contain anywhere from portions of logs to multiple logs with the log cut off from the last to the next, with the sumo logic implanted timestamps and sourceIP (always the logstash server). After that point the log is just carrying off from the last character from the previous entry and continues until some arbitrary event length and continues in the next message. Event breaks occur mid message and then just start another event in the same entry.
2. The raw log data is in JSON and at the very least i need to map the common JSON fields to the metadata fields which I understand can be done via HTTP headers.
3. The JSON format for this data is totally unclear to me.
4. There is also other key, value paired data nested in some of the messages and many of them exceed the arbitrary length of the log event in Sumo.
5. For some reason I thought I could make sense of this but I clearly can't even create valid JSON to describe the output plugin!

I tried making something like this but I don't even know if this is valid JSON:

output {
  sumologic {
    url => "https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/XXXXXXXXXXXXXXXXX"
    format => "%{@json}"
    json_mapping => {
      "_source" => "%{HOSTNAME}"
    }
  }
}

Any help would be appreciated!

[bin3377]

Thanks for being interesting to use this plug-in. format is used to define what things you want to send to sumo. As an example, you can try start with this (I assume you have logstash environment ready and know how to use json file to define output plug-ins):

output {
  sumologic {
    url => "https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/XXXX"
    format => "%{@timestamp} %{host} %{@json}"
  }
  stdout { }
}

As you can see, in format parameter, you can define a combination of fields in event (as %{@timestamp}). and %{@json} shows the whole event object in json format (e.g. the fields you can use)

If you are thinking about converting the event to output line with some complex processing, I may suggest you write your own filter plugin

[hinder90]

Thank you for your reply.

Our logstash template is very basic, much like the format you have given in your example. The question remains how to best map these to the Sumo Logic metadata fields using HTTP headers as described in the SL documentation. Looking at your parameters I see that you support the specification of "extra headers" in hash form. I presume this is a mapping of the header name to a string? For instance, could I map "X-Sumo-Host" to "%{host}"?

Thanks again.

[bin3377]

Theoretically it should work but I didn't test it as this way. Please file an issue if it's not work as expected. Thanks!