[Rule Tuning] Add `event.ingested` to Analytic ML Package Rules

[pantea-elastic]

Related Issues

• https://github.com/elastic/sdh-security-team/issues/968

Summary

All machine-learning related detection rules that are for an analytic package, listed below, need to have event.ingested added to their rule contents. This should ONLY happen after the integration updates happen for each integration. These updates will add event.ingested as a field during event generation.

#### Detection Rules Repo
- [ ] https://github.com/elastic/detection-rules/issues/3703

#### Integrations Repo
- [ ] https://github.com/elastic/integrations/pull/9947

Additional context: While enabling the rule "Statistical Model Detected C2 Beaconing Activity with High Confidence" a Warning message is shown in the screen stating that there are issues with a missing timestamp override field "event.ingested". The timestamp override field defined in the rule doesn’t exist in the managed index for this related integration and so the rule is throwing a warning out of the box.

[jmcarlock]

I was able to reproduce this issue. I'm currently working on a fix to the integration package.

[jmcarlock]

PR has been opened for the Beaconing integration package

[jmcarlock]

Merged SDH related Beaconing PR in the integrations repo. This issue is still open for the other Security ML integration packages.

[jmcarlock]

Hi @terrancedejesus and @brokensound77, I was looking at the DED, LMD, DGA, and LotL (ProblemChild) packages today and we are not currently able to change any more of the integration packages to output event.ingested.

The transform for Beaconing was able to be updated. For LMD there is a transform for RDP session times, but it is not used directly by the detection rules.

DGA and ProblemChild are enriching some log indices which will already have event.ingested. For instance command_and_control_ml_dga_activity_using_sunburst_domain.toml is working directly from a logs index and timestamp_override = "event.ingested" is present. These rules are correctly configured for DGA/ProblemChild.

The rest of the rules rely on outputs from the anomaly detectors (the .ml-anomalies- indices), which I do not believe support a fallback timestamp assignment. This would need to be supported upstream, as the timestamp that output from an AD is the start of a timestamp for a bucketed interval, IE 5 minutes. They do have a second of timestamp of log_time, the time the anomaly detector logged a result, but timestamp will always exist for .ml-anomalies- indices.

[terrancedejesus]

@jmcarlock - Thank you for the insight and information. We have also confirmed via conversation with security integrations that between the push for ecs@mappings and dynamic field generation on ingestion, that it will be difficult to validate per integration if event.ingested is used. The assumption at the moment is that event.ingested will be either statically mapped in the integration, as you have done, or dynamically generated on ingestion or via ecs@mappings. As this is the case, we have adjusted our code to do a static check within the integration, but only warn if the field is not found.

Based on your feedback, we have also reverted the changes to all ML-related rules as these do not contribute to the root cause of this issue.

Moving forward, we will decide internally some minor adjustments to our pull request for the unit test before merging in, but there will be no need to adjust the rules specifically and we can close the loop on your end.

Thank you for your help!

[jmcarlock]

@terrancedejesus You're welcome. Thank you for the update! I have marked this task as complete on our end.

[terrancedejesus]

@andrewkroh or @efd6 - Per this discussion, can you confirm or deny that statically identifying if event.ingested is a guaranteed field populated in any integration data stream is feasible after our conversation?

If not, as I understand, then I will close the pull request referenced here for development.

[andrewkroh]

As of 8.13 (with this change), Fleet will be including the ecs@mappings component template into the data streams that are managed by Fleet. This component template defines dynamic mapping rules for ECS fields including event.ingested. So packages running on >=8.13.0, its data streams should always contain a date mapping for event.ingested.

Note that I said data streams, transforms do not write to data streams. I'm not sure if Fleet extends the same guarantees to the destination indices belong to transforms. In the past we noticed many inconsistencies relating to how index templates for transforms were handled by Fleet compared to the data streams. I don't see the ecs@mappings template being applied to latest transform destination indices (see the image below). So I still think it is up to the package to ensure that it defines a mapping for event.ingested if it uses it in a transform index.

[Mikaayenson]

Might want to then consider a scoped unit test for 8.12 and below. @andrewkroh can you on the integrations side add a unit test to enforce so that package authors don't incidentally create one without the proper fields mapped?