Replace ECS static mapping with ECS dynamic mapping for AAD indices

[maryam-saeidi]

📝 Summary

In Alert-As-Data (AAD), all the ECS fields are added statically to the mapping. In this ticket, I would like to discuss if it is a good idea to change it to dynamic mapping and how it can be done. (Slack discussion)

Motivation

@P1llus mentioned:

The dynamic template takes up much less memory per node (cluster state), than the full field mappings and the searching is much much quicker when you have 100 fields rather than 2000-3000

@simianhacker mentioned that when we add ECS fields statically, many fields are added to the mapping regardless of whether they are used or not, and as a result, in the KQL suggestion bar, the user will face many fields that many of them probably are not relevant.

Challenge

The challenge is ensuring dynamic mapping will not exceed the limit set for the number of fields in alerting. Here is the limit, which seems 1900 of them are ECS fields. This limitation was discussed in this RFC as well.

Possible solution

One solution to solve this issue is using true_until_limit that will be introduced in this PR. But @pmuellr shared this concern:

The downside of that PR - using the new true_until_limit option - is that we aren't in control of what fields would end up NOT getting mapped. Which may end up being confusing, where one dynamic field could be mapped and another not. Probably even more complicated to debug if the limit is quite high, and we rarely run into the issue, and have 1000's of fields we have to investigate.

✅ Acceptance Criteria

• Based on the discussion, if it makes sense, replace ECS static mapping with ECS dynamic mapping.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[maryam-saeidi]

@ymao1 What are your thoughts about this?

[ymao1]

Definitely worth discussing! Since we'd be using the dynamic mapping in conjunction with alerts as data fields, what happens when we have a kibana.alert.* field that matches a dynamic mapping path match or falls into the all_strings_to_keywords bucket but we want a different mapping than defined for the ECS mappings? What takes precendence?

We should probably also get the opinion of @elastic/security-detections-response-alerts since the use the ECS fields most heavily. It would be good to do a POC to make sure transitioning from a static to a dynamic template on upgrade is possible and seamless.

[P1llus]

@ymao1 specific field mappings always takes priority over dynamic templates. First the field is checked to see if a mapping exists, if not the matching with dynamic template is applied.

A few things that would need to be considered is that you have dynamic:false set because you don't want to map non-ecs fields, it might be hard to run dynamic templates while also disabling this for non-ecs fields. This could result in a larger field count since the custom fields also gets a field mapping.

But im sure there are ways around it :)