sf-exporter: Buffer re-use in json encoder leads to corrupt messages

[san-zrl]

Indicate project sf-processor, json encoder

Describe the bug The json encoder in sf-processor uses a pre-allocated buffer (t.buf) to avoid multiple memory allocations. The buffer re-use happens on line 271 in sf-processor/core/exporter/encoders/json.go (currently commented out).

The exporter module in sf-processor's pipeline configuration allows to specify a buffer parameter as a batch size (number of records to be encoded/transported at once). For buffer > 1, the buffer re-use causes corrupt json output because old records will be overwritten by new ones without flushing the buffer in between.

To reproduce Steps to reproduce the behavior:

1. Change line 270-271 in sf-processor/core/exporter/encoders/json.go:

       return t.writer.BuildBytes(t.buf)
       // return t.writer.BuildBytes()

2. Re-compile
3. Run sf-processor using the pipeline configuration below

Files

{
    "pipeline":[
      {
       "processor": "sysflowreader",
       "handler": "flattener",
       "in": "sysflow sysflowchan",
       "out": "flat flattenerchan"
      },
      {
       "processor": "policyengine",
       "in": "flat flattenerchan",
       "out": "evt eventchan",
       "mode": "alert",
       "policies": "../resources/policies/runtimeintegrity/ttps.yaml"
      },
      {
       "processor": "exporter",
       "in": "evt eventchan",
       "export": "terminal",
       "format": "json",
       "buffer": "1000"
      }
    ]
}