search

SumoLogic / fluentd-output-sumologic

Fluentd output plugin to deliver logs or metrics to Sumo Logic.
https://rubygems.org/gems/fluent-plugin-sumologic_output
Apache License 2.0
29 stars 28 forks source link

Don't concatenate buffer results #64

Closed willthames closed 2 years ago

willthames commented 4 years ago

Joining records back together while batching sends causes problems for multiline parsing.

For example, with the records

2020-10-05T10:00:00Z First line of an error message
  second line of an error message
2020-10-05T10:00:00Z A different error message

if multiline parsing is set correctly, we expect this to be split into two records, the first of two lines and the second of one line. However, if records are joined at the point of sumo publishing, then we may well get all three lines as a single record.

This is mostly a problem for multiline parsing as you can treat each line as an individual message if multiline is not used, but you cannot do that if needing to do the multiline separation before sending to sumo.

sumo-drosiek commented 2 years ago

Hi @willthames,

sorry for veeery late response. This is true and in order to keep multilines as they are we recommend to use json log format.

We cannot send every log as separate message, because it's not efficient for larger environments

I believe the #78 would be used to cover that case when you would like to use text format and send every log as separate request

sumo-drosiek commented 2 years ago

I'm closing due to inactivity.