[Security Solution] When investigating Threshold alerts with a timestamp override, the timeline's `from` date should be based on the value of the override

[andrew-goldstein]

Summary

When investigating Threshold alerts with a timestamp override, the timeline's from date should be based on the value of the override, e.g. event.ingested.

The current implementation of the getThresholdAggregationData function in x-pack/plugins/security_solution/public/detections/components/alerts_table/actions.tsx does not use the value of the timestamp override when investigating Threshold alerts.

As a result, the from to part of the date range is invalid when a user investigates a Threshold rule with a timestamp override in Timeline.

See related issue: https://github.com/elastic/kibana/issues/144467

Details

For Threshold rules, the current implementation of the getThresholdAggregationData function in x-pack/plugins/security_solution/public/detections/components/alerts_table/actions.tsx, which implements the Investigate in timeline action (for Threshold rules), works as follows:

• The from time is determined by the kibana.alert.threshold_result.from field on this line in 8.4.
• The to time is determined by the kibana.alert.original_time field on this line in 8.4.
  • The business logic in the getThresholdAggregationData function does NOT use the kibana.alert.rule.timestamp_override in the alert document.
  • The value of the kibana.alert.rule.timestamp_override field in the alert document does not contain a timestamp. It contains the name of a field, e.g. event.ingested.
  • Per the details below, the event.ingested field may NOT be populated in the alert document for Threshold rules, unless it's a group by field
• The getThresholdAggregationData function doesn't have business logic to sanitize the to date, such that it is greater than or equal to the from date.

@marshallmain provided the following details regarding this issue:

When a timestamp override is used in detection rules, the detection rule uses the timestamp override field to determine if source documents are in range of the query. However, (the Investigate in timeline) code uses @timestamp with the to and from dates to filter source documents.

When a kibana.alert.rule.timestamp_override is configured for a threshold rule, is the from time expected to be based on the value of the override field, e.g. event.ingested?

The from time should generally be based on the value of the override field.

If the answer to the previous question is yes, is the Detection Engine expected to populate the kibana.alert.threshold_result.from field with the value from the kibana.alert.rule.timestamp_override field (i.e. event.ingested), or is the business logic in the getThresholdAggregationData function expected to implement branching logic when an override is configured?

Ideally getThresholdAggregationData would create a timeline that filters documents using the correct override timestamp field. kibana.alert.threshold_result.from and kibana.alert.original_time should always contain the correct time bounds for retrieving the source documents.

Some threshold alerts may not have the event.ingested field populated. When the field specified by the kibana.alert.rule.timestamp_override is not populated, what should the fallback be, (if the override field should be used to determine the from time)?

The alert document is not expected to populate event.ingested in this case. Currently for threshold alerts only the fields that were "grouped by" are copied over to the alert doc from the source documents. When querying the source documents, the detection rule query uses @timestamp as the fallback time field by default.

Should the getThresholdAggregationData function sanitize the to date, such that it is always greater than or equal to the from date?

I don't think so, I think we should fix the threshold alert generation so that the fields are created as expected.

Kibana/Elasticsearch Stack version:

8.5.0

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[yctercero]

Hey @paulewing ! Would this be threat hunting?