[m365] M365 Defender Fingerprint not able to sync changes from M365 Portal

[nicpenning]

It appears that the M365 Defender Integration is creating a fingerprint based on some lastUpdate times:

https://github.com/elastic/integrations/blob/d38087c71ce6dd66bc63ef08935f6e4e02777c38/packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml#L26-L35

This is a problem because there are multiple events/documents for a single incident and when the incident's status changes from new to resolved, only 1 of the documents is updated to reflect this change in Elastic. This will then in turn effect the dashboard and show incorrect results on active/resolved incidents. I am sure this will apply to alerts as well.

I recommend testing the removal of:

       - json.lastUpdateDateTime
       - json.alerts.lastUpdateDateTime 

To see if that resolves the issue.

I have tested this in my own environment and it seems to work well but it makes sense for the creator of this integration to check and make sure there isn't any other use cases I may be missing.

Here is part of what the issue looks like:

[nicpenning]

Looking further into this, it appears that lastUpdateTime is the @timestamp which is not accurate and misleading from my perspective. I need to know the original event creation date/time and not when it get's update. That could force these incidents to move from last week to today simply because there is another piece of information aggregated to the incident.

So I would encourage that we use the json.createdDateTime for the @timestamp, also.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

This is a problem because there are multiple events/documents for a single incident and when the incident's status changes from new to resolved, only 1 of the documents is updated to reflect this change in Elastic.

I'm confused by this comment; when there is an update, it would have a new lastUpdateTime and so would be a new document. No documents will be updated. Is that what you are concerned about?

In the screenshot, is that before you applied the change or after (I'm assuming before)?

I can see merits in both approaches; the current approach is favouring not dropping data, which is the default approach for our designs, while you are wanting a current state. The choice of origin for the @timestamp date matches the current design. Maybe we could make these conditional.

[nicpenning]

Great points.

Yes, I would like all documents, but if I ever want to see the current state, I am unsure how that is achieved. The dashboard that is built in which shows active and resolved incidents (and the alerts one too), provide false information. I would rather have less and more current and accurate information rather than an event that is no longer accurate but could have been during some point in time. I am open to how both options could be achieved.

My screenshot is a snapshot of what I believed what was before, but the take away it appears to have multiple of the same event with only a small set of fields that changed. There is a chance my update by query could have tweaked some of these results because at scale the status is worthless today in the built in dashboards because of the 10s, 100s, and sometimes even 1000s of events that are duplicates with the stale information on the status.