[Security Solution] Impossible to add a runtime field for `winlog.event_data.ServiceFilename`

maximpn commented 5 months ago

Summary

Alerts table doesn't provide a way to filter by fields without mappings, e.g. winlog.event_data.ServiceFilename. At the same time it doesn't allow to add a runtime field for winlog.event_data.ServiceFilename complaining that this field exists.

Runtime field is required to filter out or search for alerts.

Steps to reproduce

Index a source event

POST /logs-test-1/_doc
{
  "@timestamp": 1713346225559,
  "event": {
    "kind": "event",
    "category": "iam",
    "type": ["group", "info"]
  },
  "winlog": {
    "event_data": {
      "ServiceFilename": "abc"
    }
  }
}

Create a custom query rule with *:* query to "promote" all source event to alerts
Run the rule
Make sure an alert has been generated
Use Stack Management -> Data Views -> Security Data View -> Add Field or Fields button right above alerts table in grid mode -> Create field button
Try to create runtime mapping

ER: It's possible to create a runtime field for winlog.event_data.ServiceFilename. AR: Errors appear.

Entering winlog.event_data.ServiceFilename in Name field leads to A field with this name already exists. error

Entering a modified name runtime.winlog.event_data.ServiceFilename and a script

def source = doc['winlog.event_data.ServiceFilename'].value;

if (source != "") {
  emit(source);
} else {
  emit("None");
}

leads to No field found for [winlog.event_data.ServiceFilename] in mapping.

UPDATE: The following script allows to create a runtime field

if (params._source.containsKey("winlog") && params._source["winlog"].containsKey("event_data") && params._source["winlog"]["event_data"].containsKey("ServiceFilename")) {
    emit(params._source["winlog"]["event_data"]["ServiceFilename"]);
}

Additional notes

It's possible to add runtime mapping for winlog.event_data.ServiceFilename on a clear instance. In the other words until some data appears in Security Data View which maps to .alerts-security.alerts-default,apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,traces-apm*,winlogbeat-*,-*elastic-cloud-logs-*.

In this case filtering at Alerts table works.

elasticmachine commented 5 months ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine commented 5 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 5 months ago

Pinging @elastic/security-detection-engine (Team:Detection Engine)

yctercero commented 3 weeks ago

We're working on determining a good UX for supporting unmapped alert index fields. Added this ticket to the relevant epic.

elastic / kibana