matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
https://matano.dev
Apache License 2.0
1.46k stars 100 forks source link

Add a JSON/object type for semi structured data #28

Open Samrose-Ahmed opened 1 year ago

Samrose-Ahmed commented 1 year ago

Overview

Currently, semi structured data must be stringified and defined as a string type. Subsequently, it is always treated as a string type (e.g. in detections).

Goal

Add a JSON or object type that represents JSON/object data. This can be auto-coerced to physical type string for the Iceberg schema. We can use this information to provide additional features for detections and searching.

Notes