matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
https://matano.dev
Apache License 2.0
1.44k stars 97 forks source link

Bug when defining a custom log source with all custom table configurations #134

Open shaeqahmed opened 1 year ago

shaeqahmed commented 1 year ago

Bug brought up in discord channel:

Was popping in because I had a quick question. Is it possible to configure tables with a custom log source? Couldn’t seem to get that working and when walking through the code

My setup looks something like this:


# /log_sources/customsource/log_source.yml
name: customsource

ingest: select_table_from_payload_metadata: | if match(.metadata.s3.key, r'.*') { "mytable" } expand_records_from_payload: "parse_json!(.raw)"

schema: ecs_field_names:

transform: | .event.original = encode_json(.json)

/log_sources/customsource/tables/mytable.yml

name: "mytable"

schema: fields:

I expected it to create a mytable.yml schema, but instead it creates default.yml and ignores any configurations in my table.

The log-source.ts construct needs to be updated to properly pick up user configuration for custom log sources / tables.