[M365 Defender] Improve ECS mappings

[kaltman]

Security->Dashboards->Data Quality dashboard reporting incorrect ECS mapping for dns.answers ECS is expecting an object while the integration is setting keyword:

1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly ❌ Pages may not display some events or fields due to unexpected field mappings or values ❌ Mappings or field values that don't comply with ECS are not supported

Incompatible field mappings - .ds-logs-m365_defender.event-default-2024.07.13-000037

Field	ECS mapping type (expected)	Index mapping type (actual)
dns.answers	object	keyword

https://github.com/elastic/integrations/blob/2565466908adf3634efa69495e86d045834933cd/packages/m365_defender/data_stream/event/fields/fields.yml#L200-L202

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[predogma]

Issue is reported in Kibana 8.14.3 with M365 Defender integration version 2.14.2

[efd6]

I believe this is a false positive. The dns group in this case is under m365_defender.event, so it is not the ECS dns group. If we are constrained to follow the ECS for fields that have names that match ECS field names despite being in conventional locations for data stream-specific data it becomes very difficult to maintain those stream-specific data.

[predogma]

How to rectify the 'false positive' such that the M365 Defender indices work with the "Data Quality dashboard", current the response is

#### Incompatible field mappings - .ds-logs-m365_defender.event-default-2024.07.13-000037

| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| dns.answers | `object` | `keyword` |

Please advise

Is the only recourse then, to reindex these M365 created, indices? Then also, do we need to modify the integration pipeline adding custom mapping for that field to make it object? If so we would expect it may break other visualizations that belong to this integration. Assuming this is the fix, then we expect this is a bug with the integration (M365 Defender)?

[chemamartinez]

I already opened https://github.com/elastic/integrations/pull/10772 to fix how the dns.answers is handled. As it was being directly copied from m365_defender.event.dns.answers, its type (keyword) was also inherited. After the change, it should contain an object as expected.

The change should not affect any assets from the integration as none of them use the dns.answers field. However, I would say a reindex of the data from the old indices would be necessary after the upgrade to keep consistency. I think the easiest way to do this, if the data contains the event.original field, would be to reindex the data from that field through the new ingest pipeline.