search

philhagen / sof-elk

Configuration files for the SOF-ELK VM
GNU General Public License v3.0
1.49k stars 277 forks source link

Azure AD SignIn Logs Not Parsed #274

Open dsplice opened 1 year ago

dsplice commented 1 year ago

When attempting to ingest AzureAD SignIn logs downloaded from the portal in JSON, the files (EG "InteractiveSignIns_2023-05-08_2023-05-09.json") are not ingested at all.

When attempting to ingest AzureAD SignIn logs downloaded using powershell (eg "Get-AzureADAuditSignInLogs -Filter "userPrincipalName eq 'derek.armstrong@domain.com'" | ConvertTo-Json | Out-File -FilePath C:\Temp\test\dereklogin.json") the logs are ingested but without any parsing or enrichment.

The data appears different from the data released in FOR509. Not sure if they need a different way of downloading, or if the format has changed.

Pierre450 commented 1 year ago

@dsplice The problem with getting logs from PowerShell commands is that Microsoft keeps making changes and keeps threatening to deprecate these commands to force everyone to use Graph API. For Azure logs, it's much easier to send the logs to a storage account (which is what we use in FOR509).

dsplice commented 1 year ago

@Pierre450 Understood. Doesn't help much with my current engagement but will be trying the storage account method in the future.

Pierre450 commented 1 year ago

@dsplice Take a look at this project: for509.com/invictus-suite It's from Korstiaan Stam who is a FOR509 instructor. He has done a lot of work to make sure to extract the logs in a way that can be imported into SOF-ELK. In the example you gave, it's highly likely that "ConvertTo-Json" cmdlet is doing some strange things. When I was testing with Korstiaan, we had to make sure to force UTF8 or the files wouldn't import.

dsplice commented 1 year ago

I gave Invictus-Suite a try. It uses the command I listed in the original issue. I tried importing the resulting file, and the same outcome. It would only parse it generically. No fields brought out, timestamp was all set to the extraction time. So I do not think it is the ConvertTo-Json that is causing the issues, but will continue to dig.

hutspot79 commented 1 year ago

I have the same problem. All my Microsoft 365 logs, both in CSV and JSON format, are only processed globally. The content log fields are not processed and displayed in Kibana. The date and time were set to the moment of adding them to SOF-ELK. I used the Invictus extractor suite to export the logs via Powershell. I have written the logs directly as JSON files. Are there any options left to try?

dsplice commented 1 year ago

I saw that the files I was trying to play with were in UTF-8 when the signin logs from Invictus were in UTF-16. But since the parser is forcing UTF-8, I doubt that should be an issue. I did notice a very different amount of data coming from Invictus vs other methods, plus key names in the file appear to have different capitalizations. I might look at seeing what it will take to create a logstash config to parse it through, just need time.

philhagen commented 11 months ago

I suspect this is related to #285. If you can send me a sample of a problem file, I can dig in

mdenzel commented 9 months ago

I can confirm this issue. Unfortunately, I cannot provide sample data as it is not my data. However, I tried all of the following and importing always failed:

spivjohn commented 2 months ago

@philhagen I see other people are having the same issue that I experienced today. I have exported the Azure SignIn logs using the Invictus-Suite Get-ADSignInLogsGraph. This as you know creates output files in JSON format but when they are ingested into Sof-ElK they are not parsed correctly. Is this still a known issue with no fix as yet?

dabltrabl commented 15 hours ago

Hello all do we have some updates on this issue? @philhagen Ive tested the parsers from last image and still have the same problems Im interested in writing this parses on my own in future on additional log samples so if you're available to share some details and thoughts or some more information on these it would be largely beneficial. Even the time you would work on new one would be excellent to see. Ive seen your comments on currently available parsers and they give some insight but its hard to start from 0 without some more context. Ill start tomorrow so i hope i will share the successful result soon... I can share a sample file just contact me, i have made a lab for this use case. BR

philhagen commented 15 hours ago

Apologies. Been a very busy few weeks but after the new VM released, I should have some bandwidth to look at this again.

Couple steps:

With answers to those, I can start to dig in. However, in discussion with @Pierre450, and looking at FOR509 lab material in particular, we've had trouble reproducing it.