brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
330 stars 21 forks source link

feat: Add JSON Lines Support to KV Store #126

Closed jshlbrd closed 1 year ago

jshlbrd commented 1 year ago

Description

Motivation and Context

GJSON has native support for JSON Lines and this is the simplest way to support using them as a KV store.

How Has This Been Tested?

Integration tested using the development app. The test can be recreated with these configs:

kv.json

{"a":1}
{"b":2}
{"c":3}
{"d":4}
{"e":5}

process.libsonnet


local sub = import '../../build/config/substation.libsonnet';

local processors = [ { processor: sub.interfaces.processor.kv_store( options={ type: 'get', kv_options: sub.interfaces.kv_store.json_file( settings={file: 'kv.json', is_lines: true} ) }, settings={key: 'bar', set_key: 'kv_store'} ) }, ];

{ processors: sub.helpers.flatten_processors(processors), }


> data.json
```json
{"foo":"a"}
{"bar":"b"}
{"baz":"c"}
{"qux":"d"}
{"quux":"e"}

The results are:

{"foo":"a","kv_store":null}
{"bar":"b","kv_store":2}
{"baz":"c","kv_store":null}
{"qux":"d","kv_store":null}
{"quux":"e","kv_store":null}

Types of changes

Checklist: