Columns format in Defender for Identity tables has changed over time

[ep3p]

Describe the bug Tables from Defender for Identity solution (IdentityDirectoryEvents, IdentityLogonEvents, IdentityQueryEvents) have changed their format for some columns in the past months.

To Reproduce Steps to reproduce the behavior:

1. Go to a Sentinel workspace where the Microsoft 365 Defender connector is enabled, and it is receiving events in table IdentityQueryEvents.
2. Go to Logs tab.
3. Execute the following query IdentityQueryEvents | where Protocol has "Ldap" | project Query | limit 1000
4. Observe the column Query has " characters.
5. Go to a Microsoft 365 Defender portal that it is receiving events in table IdentityQueryEvents.
6. Go to Advanced Hunting tab.
7. Execute the following query IdentityQueryEvents | where Protocol has "Ldap" | project Query | limit 1000
8. Observe the column Query does NOT have " characters.
9. Compare both columns and realize they do not have the same format.
10. The same thing happens with Protocol column in IdentityLogonEvents ("NTLM" value does not have the same case).

Expected behavior Both tables should have the same format for all columns. I had developed analytics rules in Sentinel for Defender for Identity tables (the Query column did not have " characters) and they stopped getting results because of this unexpected change. I believe this change might be related to when the old Defender for Identity portal was migrated to Microsoft 365 Defender portal.

Screenshots

Thank you for assisting me.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-sudkharat]

Hi @ep3p, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 16-10-2023 . Thanks!

[v-sudkharat]

Hi @ep3p, hope you are doing well. Could you please share the list of analytic rules which stopped getting results? due to those columns. It would be helpful to us to resolve this issue. Thanks!

[ep3p]

I had developed analytics rules in Sentinel for Defender for Identity tables (the Query column did not have " characters) and they stopped getting results because of this unexpected change.

@v-sudkharat the analytics rules I developed are custom, so you don't have to worry about them, I have not observed Analytics Rules in this repo (https://github.com/Azure/Azure-Sentinel/) that might use IdentityQueryEvents table.

If you are curious, this query returned results (in Sentinel) some months ago:

IdentityQueryEvents
| where Protocol == "Ldap"
| parse Query with "LDAP Search Scope: " SearchScope ", Base Object:" BaseObject ", Search Filter: " SearchFilter
| where SearchScope == "WholeSubtree"

but it doesn't anymore, because the value for SearchScope column was parsed as WholeSubtree, and currently it is parsed as "WholeSubtree", so the == should be substituted by something like has (we could also use == '"WholeSubtree"'):

IdentityQueryEvents
| where Protocol == "Ldap"
| parse Query with "LDAP Search Scope: " SearchScope ", Base Object:" BaseObject ", Search Filter: " SearchFilter
| where SearchScope has "WholeSubtree"

The query where I compared SearchScope was this one, so at least in Oct 19, 2022 the column Query in IdentityQueryEvents in Sentinel did NOT have " characters.

Thank you for assisting me.

[v-sudkharat]

Hi @ep3p, thanks for response. we are investigating this issue, and we will get back to you with some updates by 25-10-2023. Thanks!

[v-sudkharat]

Hi @ep3p, we received an update from concern team, they are working on revert the changes for data sources. Once we receive further details from them, we'll let you know. ETA -31-10-2023 Thanks!

[ep3p]

@v-sudkharat thank you very much for your help :)

[v-sudkharat]

Hi @ep3p, we are following with the concern team for this issue, once we get any information on this, we will update you. Thanks!

[ep3p]

@v-sudkharat thank you again, don't worry for me, I won't need frequent updates, I understand these are complex issues and might take a lot of time.

[v-sudkharat]

Hi @ep3p, We appreciate your understanding and cooperation. Our team is working on this issue, and once we receive an update from them, we will keep you informed.

Thanks!

[v-sudkharat]

Hi @ep3p, I hope this message finds you well. We are currently coordinating with the concerned team regarding the issue and are in the process of verifying data source alignment. In the meantime, could you please check from your end is this issue has been resolved? and please let us know. Thank you!

[ep3p]

@v-sudkharat thank you very much.

Now I see the column Query, both in 365 Defender and Sentinel, has " characters. They have the same format.

I have to notice that, before these changes, NONE of the columns in both places HAD " characters.

Then Sentinel added the " characters, and now 365 Defender has added the " characters.

If these two changes is what Microsoft intended, then it is okay for me and you can consider this issue solved, the columns in both places have the same format.

BUT Microsoft has to realize that, the custom queries or rules that were developed for 365 Defender > Advanced Hunting that used this column might have stopped working (in the same way that my Sentinel rule stopped working). Maybe the clients need a notification (I don't know if there has been one).

Thank you very much again.

[v-sudkharat]

Hi @ep3p, Thank you for your confirmation and for sharing valuable feedback with us. We will share this feedback with our concerned team. Closing this issue. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.