[Security Solution] Duplicating OOTB Prebuilt Security Rules for ES|QL Require Query Metadata

[terrancedejesus]

Describe the bug: Brought to our attention via @aarju. Duplicating out-of-the-box (OOTB) prebuilt security rules that use ES|QL language fail because they require metadata to be included in the query. This metadata is not included in the queries OOTB by rule authors, however, install correctly and execute correctly without it. This issue happens when attempting to create a rule as well if no STATS is used.

Note, we also noticed this is an error when manually creating rules in the UI as well.

Example Rule: AWS IAM AdministratorAccess Policy Attached to Role

Additional Information:

• OOTB, these prebuilt security rules execute correctly as identified in personal cloud stack as well as global alert telemetry we are getting on them from customer stacks

Error:

Queries that don’t use the STATS...BY function (non-aggregating queries) must include the "metadata _id, _version, _index" operator after the source command. For example: FROM logs* metadata _id, _version, _index. In addition, the metadata properties (_id, _version, and _index) must be returned in the query response.

Kibana/Elasticsearch Stack version: This is not exhaustive testing, these are just the stacks I have checked on.

8.14.2 8.13.2 8.15.1

Browser and Browser OS versions: Google Chrome (Version 128.0.6613.138 (Official Build) (arm64))

Elastic Endpoint version: This is not exhaustive testing, these are just the stacks I have checked on.

8.14.2 8.13.2 8.15.1

Original install method (e.g. download page, yum, from source, etc.): Cloud stack deployment.

Functional Area (e.g. Endpoint management, timelines, resolver, etc.): Kibana Detection Engine Rules Management

Steps to reproduce:

Duplicating Rule

1. In a stack 8.13+, install the OOTB prebuilt rules integration.
2. Install all of the prebuilt rules under Security > Rules > Detection Rules (SIEM)
3. Search for AWS IAM AdministratorAccess Policy Attached to Role rule
4. Select "Duplicate rule" and then "Save Changes" -> Error should appear as shown in the screenshot above.

Creating Rule

1. In a stack 8.13+, install the OOTB prebuilt rules integration.
2. Under Security > Rules > Detection Rules (SIEM) > Create Rule
3. Grab query from AWS IAM AdministratorAccess Policy Attached to Role rule > Select ES|QL > Paste query in custom query box

Current behavior:

Expected behavior: This depends on what is required for the Kibana parser for ES|QL. They run fine OOTB as seen in my custom stack and customer alerts we gather, but the Kibana ES|QL parser may be using outdated grammar?

Screenshots (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

• In short, we can update all of our rule's to include the metadata and write a unit test to enforce it, however, there appears to be some disconnect between ES|QL parser requirements in the UI and what is allowed via the API and ES.
• Also, when attempting to save changes, even though it fails, the rule still saves.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[terrancedejesus]

cc @approksiu

[terrancedejesus]

@maximpn - Any initial thoughts on this? At the moment it is a blocker for any customer that wants to duplicate a rule and may align with your team visiting rule mutability subject.

[maximpn]

Hi @terrancedejesus,

I discussed this bug with Detection Engine team. The _id, _index and _version are used for deduplication, without those fields alerts may be duplicated on subsequent rule runs. We should ideally update the prebuilt rules to include those fields and add validation to prevent prebuilt rules from missing those fields in the future. We have a dedicated section to using metadata operator with such queries.

FYI @marshallmain @vitaliidm

[Mikaayenson]

👋 We just released the latest update that includes the metadata. If you update the rules, this should be resolved.