ECS compatibility in ML Anomaly indices

[dhurley14]

Description

The security solution recently received an issue where an ECS field was incorrectly mapped as a keyword, when it should be of type ip. I am opening this issue to start a conversation on if / how we can get the ml anomaly indices up to date with ECS mappings. My understanding is that these indices pre-date the development of ECS so hopefully this can be a straightforward change.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[Oddly]

Can we also start a discussion on adding the *.port to the ecs@mappings component template?

[sodhikirti07]

I discussed this with the StackML team, who handle the anomaly job indices, and received the following feedback from Sophie:

Fields added to the .ml-anomalies-* results index are all stored as keyword. This allows us to share results indices between jobs. Adding runtime fields is a viable option on a case-by-case basis, however could cause conflicts in some cases where multiple jobs share the same results index. ML results are hidden indices. They are designed for analysing arbitrary data and for any variety of job config defined by customers. The data format supports the UI. Converting ML jobs to be ECS only would be a significant effort and incur BWC complications due to potential index mapping conflicts (as well as increasing overall index/shard counts)

@pantea-elastic @dhru42 Given this information, how would you like to proceed?

[pantea-elastic]

@dhru42 this effort will have to be done on StackML side as Kirti pointed out. We will need to engage with their Product team if this is a priority.