search

elastic / detection-rules

https://www.elastic.co/guide/en/security/current/detection-engine-overview.html
Other
1.98k stars 505 forks source link

[FR] CI Job to Sync ES|QL Custom Fields with Prebuilt Filterlist for Telemetry #4168

Open terrancedejesus opened 1 month ago

terrancedejesus commented 1 month ago

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

At the moment, when using ES|QL for writing detection rule queries, often we use aggregate functions, eval or pre-processing functions (grok and dissect) to create useful fields for our filters.

In this instance, those fields are not available in global alert telemetry, which relies on a static filterlist for determining what fields to ingest from the alerts.

Desired Solution

As such, we must develop a CI job that loads the ES|QL rules, reviews custom fields and adds those to the filterlist. The CI job would run on merges into main only.

Considered Alternatives

No alternatives considered. This suggestion is post conversation with Security Data Analytics team.

Additional Context

No response

Mikaayenson commented 1 month ago

We will need to consider how to handle the internal list in CI. We may want to kickoff a job that runs in CI in a different repo.