Azure AD SignIn Logs Not Parsed

[dsplice]

When attempting to ingest AzureAD SignIn logs downloaded from the portal in JSON, the files (EG "InteractiveSignIns_2023-05-08_2023-05-09.json") are not ingested at all.

When attempting to ingest AzureAD SignIn logs downloaded using powershell (eg "Get-AzureADAuditSignInLogs -Filter "userPrincipalName eq 'derek.armstrong@domain.com'" | ConvertTo-Json | Out-File -FilePath C:\Temp\test\dereklogin.json") the logs are ingested but without any parsing or enrichment.

The data appears different from the data released in FOR509. Not sure if they need a different way of downloading, or if the format has changed.

[Pierre450]

@dsplice The problem with getting logs from PowerShell commands is that Microsoft keeps making changes and keeps threatening to deprecate these commands to force everyone to use Graph API. For Azure logs, it's much easier to send the logs to a storage account (which is what we use in FOR509).

[dsplice]

@Pierre450 Understood. Doesn't help much with my current engagement but will be trying the storage account method in the future.

[Pierre450]

@dsplice Take a look at this project: for509.com/invictus-suite It's from Korstiaan Stam who is a FOR509 instructor. He has done a lot of work to make sure to extract the logs in a way that can be imported into SOF-ELK. In the example you gave, it's highly likely that "ConvertTo-Json" cmdlet is doing some strange things. When I was testing with Korstiaan, we had to make sure to force UTF8 or the files wouldn't import.

[dsplice]

I gave Invictus-Suite a try. It uses the command I listed in the original issue. I tried importing the resulting file, and the same outcome. It would only parse it generically. No fields brought out, timestamp was all set to the extraction time. So I do not think it is the ConvertTo-Json that is causing the issues, but will continue to dig.

[hutspot79]

I have the same problem. All my Microsoft 365 logs, both in CSV and JSON format, are only processed globally. The content log fields are not processed and displayed in Kibana. The date and time were set to the moment of adding them to SOF-ELK. I used the Invictus extractor suite to export the logs via Powershell. I have written the logs directly as JSON files. Are there any options left to try?

[dsplice]

I saw that the files I was trying to play with were in UTF-8 when the signin logs from Invictus were in UTF-16. But since the parser is forcing UTF-8, I doubt that should be an issue. I did notice a very different amount of data coming from Invictus vs other methods, plus key names in the file appear to have different capitalizations. I might look at seeing what it will take to create a logstash config to parse it through, just need time.

[philhagen]

I suspect this is related to #285. If you can send me a sample of a problem file, I can dig in

[mdenzel]

I can confirm this issue. Unfortunately, I cannot provide sample data as it is not my data. However, I tried all of the following and importing always failed:

• used Invictus-Suite (json output)
• used the Powershell commands of Invictus-Suite manually (json output)
• used Powershell commands to output csv
• converted the json output into UTF-8
• converted the json output into UTF-16

[spivjohn]

@philhagen I see other people are having the same issue that I experienced today. I have exported the Azure SignIn logs using the Invictus-Suite Get-ADSignInLogsGraph. This as you know creates output files in JSON format but when they are ingested into Sof-ElK they are not parsed correctly. Is this still a known issue with no fix as yet?