brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
330 stars 21 forks source link

feat: Add SetKey Support to Expand Processor #81

Closed jshlbrd closed 1 year ago

jshlbrd commented 1 year ago

Description

Motivation and Context

Some data sources have complex object structures with either deeply nested arrays of objects or multiple layers of arrays of objects. For example:

{
    "a": {
        "b": {
            "c": [{
                "d": "e"
            }],
            "h": {
                "i": 1
            }
        }
    },
    "j": {
        "k": 1
    }
}

The processor currently pushes { "d": "e" } to the top of the object like this:

{
    "d": "e",
    "a": {
        "b": {
            "h": {
                "i": 1
            }
        }
    },
    "j": {
        "k": 1
    }
}

This makes it nearly impossible to modify the structure of the object after the processor runs because the content of the array is dynamic. This change adds SetKey support which makes this possible (if the SetKey is "x.y"):

{
    "a": {
        "b": {
            "h": {
                "i": 1
            }
        }
    },
    "j": {
        "k": 1
    },
    "x": {
        "y": {
            "d": "e"
        }
    }
}

This has some other benefits, like making the expansion of non-object arrays possible.

How Has This Been Tested?

Refactored and added new unit tests.

Types of changes

Checklist: