Closed angry-bender closed 1 week ago
angry-bender
commented
4 months ago Just looking through, Im unsure as to the tenancy sizes tested for v2.0, but if we're using the Convert-ToJSON feature of powershell, its likely to break for large tenancies when converting.
Since you area already asking for a graph request for -ContentType 'appliation/json' is likely the convert-tojson -depth 100 inst required. This might help to fix both this issue and #76
JoeyInvictus
commented
4 months ago Hmm, thanks for letting us know. I currently don't have access to a large environment to test something like this. As far as I know, there's no way to split the scan. I'm pretty sure you can only run one at a time. It might be possible to split it up by running it for a day or a week, but not in an automated way, like with the search-unifiedauditlog cdmlet.
However, the results should be paginated using the @odata.nextLink mechanism. This means it shouldn't load all results into memory before converting to JSON. Instead, each page gets converted to JSON and then written to an output file. Therefore, I'm not sure if a large tenant would cause this problem, since retrieving the results and downloading them are done in separate steps.
Code responsible for downloading the results:
$apiUrl = "https://graph.microsoft.com/beta/security/auditLog/queries/$scanId/records"
Do {
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json'
if ($response.value) {
$filePath = Join-Path -Path $OutputDir -ChildPath $outputFilePath
$response.value | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath -Append -Encoding $Encoding
} else {
Write-logFile -Message "[INFO] No results matched your search." -color Yellow
}
$apiUrl = $response.'@odata.nextLink'
} While ($apiUrl)
The combination of -ContentType and ConvertTo-Json seems redundant. I'll need to test this further to see if I can reproduce the error. In the smaller environments I have access to, I haven't encountered this issue yet. However, those environments are small...
angry-bender
commented
4 months ago Hmm, thanks for letting us know. I currently don't have access to a large environment to test something like this. As far as I know, there's no way to split the scan. I'm pretty sure you can only run one at a time. It might be possible to split it up by running it for a day or a week, but not in an automated way, like with the
search-unifiedauditlogcdmlet.However, the results should be paginated using the
@odata.nextLinkmechanism. This means it shouldn't load all results into memory before converting to JSON. Instead, each page gets converted to JSON and then written to an output file. Therefore, I'm not sure if a large tenant would cause this problem, since retrieving the results and downloading them are done in separate steps.Code responsible for downloading the results:
$apiUrl = "https://graph.microsoft.com/beta/security/auditLog/queries/$scanId/records" Do { $response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json' if ($response.value) { $filePath = Join-Path -Path $OutputDir -ChildPath $outputFilePath $response.value | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath -Append -Encoding $Encoding } else { Write-logFile -Message "[INFO] No results matched your search." -color Yellow } $apiUrl = $response.'@odata.nextLink' } While ($apiUrl)The combination of -ContentType and ConvertTo-Json seems redundant. I'll need to test this further to see if I can reproduce the error. In the smaller environments I have access to, I haven't encountered this issue yet. However, those environments are small...
Hey @JoeyInvictus Just sent through a potential fix after some testing. The only downside i can see is that it will create a significantly large amount of json files, but it might solve the issues powershell creates :-)
Calvindd2f
commented
4 months ago I've not tested it thoroughly but you can explicitly declare output type with Invoke-MgGraphRequest using Invoke-MgGraphRequest -OutputType Json. The other options for this parameter are HashTable,HttpResponseMessage,Json,PSObject for reference.
angry-bender
commented
4 months ago I've not tested it thoroughly but you can explicitly declare output type with
Invoke-MgGraphRequestusingInvoke-MgGraphRequest -OutputType Json. The other options for this parameter areHashTable,HttpResponseMessage,Json,PSObjectfor reference.
I was comparing that last night, the returned object seems to mostly be the same as -ContentType 'application/json' from what I could see, with the same type issues in output.
I wonder if it's being caused by the UAL's time data being in ISO time, regardless that option brings us back to having to specify a schema like we do on the legacy requests, which is a pain if Microsoft changes something.
We'd also need to find a way to split larger requests out to disk to avoid memory issues of we keep piping the request into a variable before outputting to file.
Simplest solution (pr #81) I've found is to use the outputFile in the invoke request, but we need to increase the chunk size, else you end up with 3,500 smaller JSON files. It also outputs that header. unfortunatley i had to close that as it also causes issues after a while
invoke-MgGraphRequest : There were not enough free threads in the ThreadPool to complete the operation.
At line:4 char:25
+ ... $response = invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentTy ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Invoke-MgGraphRequest], InvalidOperationException
+ FullyQualifiedErrorId : NotSpecified,Microsoft.Graph.PowerShell.Authentication.Cmdlets.InvokeMgGraphRequestIt seems that the $top command isn't allowing us to increase the number of results returned per page. The current limit of 150 results feels quite low to me.
Calvindd2f
commented
3 months ago Trying to work with the continuation token that is returned when the output type is HTTPResponseCode.
Also trying to parse .results.asyncstring() to a bytearray to see if it's any better for memory. After bytearray is written it is then converted to UTF8 JSON formatted.
I initially tried heavy using buffers to stream the response into chunks into the output file but it was not as easy of a fix as I thought.
Here is an over commented with printing debug statements in the function.
(begin)Here's a rundown of how it works: It connects to Microsoft Graph. Itchecks output directorythen creates it if it doesn't exist.
(begin)it makes request for the ual query and POSTs to Graph to initiate a new UAL search. It waits foq the search to start and monitors its status until it is complete.
(process)Once the search is finished, the function moves on to collecting the scan results (process) It opens file calls data. It verifies successful and increments the line count. When ln count reaches chunk size, it closes curent file and opens a new one. (For MergeOutput later)
(process)This continues until there are no more pages. When exhausted it logs a completion message.
Finally it calculates and logs the total runtime.
If any errors occur , it logs+throws the error. Finally, it ensures the StreamWriter is closed properly.
Function Get-UALGraph
{
<#
.SYNOPSIS
Gets all the unified audit log entries.
.DESCRIPTION
Makes it possible to extract all unified audit data out of a Microsoft 365 environment.
The output will be written to: Output\UnifiedAuditLog\
.PARAMETER UserIds
UserIds is the UserIds parameter filtering the log entries by the account of the user who performed the actions.
.PARAMETER StartDate
startDate is the parameter specifying the start date of the date range.
Default: Today -90 days
.PARAMETER EndDate
endDate is the parameter specifying the end date of the date range.
Default: Now
.PARAMETER OutputDir
OutputDir is the parameter specifying the output directory.
Default: Output\UnifiedAuditLog
.PARAMETER Encoding
Encoding is the parameter specifying the encoding of the CSV/JSON output file.
Default: UTF8
.PARAMETER RecordType
The RecordType parameter filters the log entries by record type.
Options are: ExchangeItem, ExchangeAdmin, etc. A total of 236 RecordTypes are supported.
.PARAMETER Keyword
The Keyword parameter allows you to filter the Unified Audit Log for specific keywords.
.PARAMETER Service
The Service parameter filters the Unified Audit Log based on the specific services.
Options are: Exchange,Skype,Sharepoint etc.
.PARAMETER Operations
The Operations parameter filters the log entries by operation or activity type. Usage: -Operations UserLoggedIn,MailItemsAccessed
Options are: New-MailboxRule, MailItemsAccessed, etc.
.PARAMETER IPAddress
The IP address parameter is used to filter the logs by specifying the desired IP address.
.PARAMETER SearchName
Specifies the name of the search query. This parameter is required.
.EXAMPLE
Get-UALGraph -searchName Test
Gets all the unified audit log entries.
.EXAMPLE
Get-UALGraph -searchName Test -UserIds Test@invictus-ir.com
Gets all the unified audit log entries for the user Test@invictus-ir.com.
.EXAMPLE
Get-UALGraph -searchName Test -startDate "2024-03-10T09:28:56Z" -endDate "2024-03-20T09:28:56Z" -Service Exchange
Retrieves audit log data for the specified time range March 10, 2024 to March 20, 2024 and filters the results to include only events related to the Exchange service.
.EXAMPLE
Get-UALGraph -searchName Test -startDate "2024-03-01" -endDate "2024-03-10" -IPAddress 182.74.242.26
Retrieve audit log data for the specified time range March 1, 2024 to March 10, 2024 and filter the results to include only entries associated with the IP address 182.74.242.26.
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]$searchName = "ExtractorSUite",
[string]$OutputDir = "Output\UnifiedAuditLog\",
[string]$Encoding = "UTF8",
[string]$startDate = (Get-Date).AddDays(-3).ToString('o'),
[string]$endDate = (Get-Date).tostring('o'),
[string[]]$RecordType = @(),
[string]$Keyword = "",
[string]$Service = "",
[string[]]$Operations = @(),
[string[]]$UserIds = @('c@lvin.ie'),
[string[]]$IPAddress = @()
)
begin
{
$authType = Get-GraphAuthType
if ($authType -eq "Delegated")
{
Connect-MgGraph -Scopes AuditLogsQuery.Read.All > $null
}
if (!(test-path $OutputDir))
{
write-logFile -Message "[INFO] Creating the following directory: $OutputDir"
New-Item -ItemType Directory -Force -Name $OutputDir > $null
}
else
{
if (Test-Path -Path $OutputDir)
{
write-LogFile -Message "[INFO] Custom directory set to: $OutputDir"
}
else
{
write-Error "[Error] Custom directory invalid: $OutputDir exiting script" -ErrorAction Stop
write-LogFile -Message "[Error] Custom directory invalid: $OutputDir exiting script"
}
}
$script:startTime = Get-Date
StartDate
EndDate
write-logFile -Message "[INFO] Running Get-UALGraph" -Color "Green"
$body = @{
"@odata.type" = "#microsoft.graph.security.auditLogQuery"
displayName = $searchName
filterStartDateTime = $script:startDate
filterEndDateTime = $script:endDate
recordTypeFilters = $RecordType
keywordFilter = $Keyword
serviceFilter = $Service
operationFilters = $Operations
userPrincipalNameFilters = $UserIds
ipAddressFilters = $IPAddress
objectIdFilters = @()
administrativeUnitIdFilters = @()
status = ""
} | ConvertTo-Json
# Open zreamWriter
$streamWriter = [System.IO.StreamWriter]::new((Join-Path -Path $outputDir -ChildPath $outputFilePath))
}
process
{
try
{
$response = Invoke-MgGraphRequest -Method POST -Uri "https://graph.microsoft.com/beta/security/auditLog/queries" -Body $body -ContentType "application/json" -OutputType HttpResponseMessage
# Convert the byte array to UTF8
$bytes = $response.Content.ReadAsByteArrayAsync().Result;
$jsonResult = [System.Text.Encoding]::UTF8.GetString($bytes);
$id = ($jsonResult | ConvertFrom-Json).id;
$scanId = $id
write-logFile -Message "[INFO] A new Unified Audit Log search has started with the name: $searchName and ID: $scanId." -Color "Green"
Start-Sleep -Seconds 4
$apiUrl = "https://graph.microsoft.com/beta/security/auditLog/queries/$scanId"
write-logFile -Message "[INFO] Waiting for the scan to start..."
do
{
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json' -OutputType HttpResponseMessage
$status = $response.IsSuccessStatusCode
if ($status)
{
$lastStatus = $status
}
Start-Sleep -Seconds 3
} while (-not $status)
if (!$status)
{
write-logFile -Message "[INFO] Unified Audit Log search has started... This can take a while..."
do
{
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json' -OutputType HttpResponseMessage
$status = $response.IsSuccessStatusCode
if (!$status)
{
write-logFile -Message "[INFO] Unified Audit Log search is still running. Waiting..."
$lastStatus = $status
}
Start-Sleep -Seconds 5
} while (!$status)
}
write-logFile -Message "[INFO] Unified Audit Log search complete."
}
catch
{
write-logFile -Message "[INFO] Ensure you are connected to Microsoft Graph by running the Connect-MgGraph -Scopes 'AuditLogsQuery.Read.All' command before executing this script" -Color "Yellow"
Write-logFile -Message "[ERROR] An error occurred: $($_.Exception.Message)" -Color "Red"
throw
}
end {
#region results
# Microsoft Graph call to get the results of invoked scan.
try
{
write-logFile -Message "[INFO] Collecting scan results from api (this may take a while)"
$date = [datetime]::Now.ToString('yyyyMMddHHmmss')
$outputFilePath = "$($date)-$searchName-UnifiedAuditLog.json"
$apiUrl = "https://graph.microsoft.com/beta/security/auditLog/queries/$scanId/records"
$chunkSize = 100000 # 100,000 records per file
$lineCount = 0
$fileCount = 0
$currentFile = New-Object -TypeName System.IO.StreamWriter -ArgumentList ($outputFilePath + "_$fileCount")
Do
{
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json' -OutputType HttpResponseMessage -OutputFile $currentFile.BaseStream
if ($response.IsSuccessStatusCode)
{
$lineCount += $response.Headers."x-ms-record-count"
if ($lineCount -ge $chunkSize)
{
$currentFile.Close()
$currentFile = New-Object -TypeName System.IO.StreamWriter -ArgumentList ($outputFilePath + "_$($fileCount + 1)")
$lineCount = 0
$fileCount++
}
# Check for the next page link
$apiUrl = $response.Headers.'@odata.nextLink'
}
else
{
Write-logFile -Message "[INFO] No results matched your search." -color Yellow
$apiUrl = $null
}
} While ($apiUrl)
write-logFile -Message "[INFO] Audit log records have been saved to $outputFilePath" -Color "Green"
$endTime = Get-Date
$runtime = $endTime - $script:startTime
write-logFile -Message "[INFO] Total runtime (HH:MM:SS): $($runtime.Hours):$($runtime.Minutes):$($runtime.Seconds)" -Color "Green"
}
catch
{
Write-logFile -Message "[ERROR] An error occurred: $($_.Exception.Message)" -Color "Red"
throw
}
finally
{
# Ensure the StreamWriter is closed
$currentFile.Close()
}
}
#endregion
#region Cleanup
#endregion
}
}Something was going wrong with the above so I tried splitting out the functions into smaller ones to later implement as begin,process,end:
# This function is responsible for creating a new Unified Audit Log search.
# It sends a POST request to the Graph API with the necessary parameters and returns the scan ID and API URL.
Function Create-UALSearch
{
# Write a log message indicating that the request to start the search has been sent
write-logFile -Message "[INFO] Starting Create-UALSearch function..."
# Send a POST request to the Graph API to create a new UAL search
# The URI is the endpoint for creating a new UAL search
# The Body contains the necessary parameters for the search
# The ContentType is set to application/json as the request expects JSON data
# The OutputType is set to HttpResponseMessage to capture the response from the API
write-logFile -Message "[INFO] Sending request to start Unified Audit Log search..."
$response = Invoke-MgGraphRequest -Method POST -Uri "https://graph.microsoft.com/beta/security/auditLog/queries" -Body $body -ContentType "application/json" -OutputType HttpResponseMessage
# Write a log message indicating that the request was successful and the response was received
write-logFile -Message "[INFO] Request sent successfully. Response received."
# Convert the response content to a string using UTF8 encoding
# The response content is a byte array so it needs to be converted to a string
# The response from the API is a JSON object with an ID property
# The ID is extracted from the response and stored in the $id variable
$bytes = $response.Content.ReadAsByteArrayAsync().Result;
$jsonResult = [System.Text.Encoding]::UTF8.GetString($bytes);
write-logFile -Message "[DEBUG] Response content: $jsonResult"
$id = ($jsonResult | ConvertFrom-Json).id;
write-logFile -Message "[INFO] Extracted scan ID: $id"
# Store the scan ID in the $scanId variable
$scanId = $id
# Write a log message indicating that a new UAL search has been started with the provided name and scan ID
write-logFile -Message "[INFO] A new Unified Audit Log search has started with the name: $searchName and ID: $scanId." -Color "Green"
# Wait for 4 seconds to allow the search to start
Start-Sleep -Seconds 4
# Set the API URL for querying the UAL search results
$apiUrl = "https://graph.microsoft.com/beta/security/auditLog/queries/$scanId"
write-logFile -Message "[INFO] API URL for querying UAL search results: $apiUrl"
# Wait for 15 seconds to allow the search to start
write-logFile -Message "[INFO] Waiting for the scan to start..."
Start-Sleep 15
# Return the scan ID and API URL as a tuple
write-logFile -Message "[INFO] Returning scan ID and API URL from Create-UALSearch function."
return $scanId, $apiUrl
}# This function is responsible for fetching records from the audit search
# The function will continue to fetch records until there are no more records to fetch
# It will return a custom object with the following properties:
# - value: The entire response from the API
# - records: The actual records extracted from the response
# - meta: Metadata about the response including skiptoken, record count, and total count
Function Get-Records
{
# Step 1: Pull the audit search status to get the URL for the records
Write-Host "[INFO] Pulling audit search status: $apiUrl" -ForegroundColor Green
Write-Host "[DEBUG] API URL: $apiUrl" -ForegroundColor DarkYellow
$apiUrl = "https://graph.microsoft.com/beta/security/auditLog/queries/$scanId/records"
# Step 2: Send a GET request to the records API URL
Write-Host "[INFO] Sending request to $apiUrl" -ForegroundColor Green
Write-Host "[DEBUG] API URL: $apiUrl" -ForegroundColor DarkYellow
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json' -OutputType HttpResponseMessage
Write-Host "[INFO] Request sent successfully. Response received." -ForegroundColor Green
# Step 3: Convert the response content to a JSON string and parse it into a PSObject
# Convert the byte array to UTF8
$bytes = $response.Content.ReadAsByteArrayAsync().Result;
$jsonResult = [System.Text.Encoding]::UTF8.GetString($bytes);
Write-Host "[DEBUG] Response content: $jsonResult" -ForegroundColor DarkYellow
$result = $jsonResult | ConvertFrom-Json
# Step 4: Check if there are more records to fetch
$nextLink = $result.'@odata.nextlink'
# Step 5: Create a custom object with the required properties
$result = [PSCustomObject]@{
value = $response
records = $result
meta = [PSCustomObject]@{
skiptoken = $skiptoken;
recordcount = $result.count;
totalcount = $result.totalcount;
}
}
# Step 6: Loop until there are no more records to fetch
while ($nextLink) {
# Step 7: Send a GET request to the next link
Write-Host "[INFO] Fetching next link: $nextLink" -ForegroundColor Green
Write-Host "[DEBUG] API URL: $nextLink" -ForegroundColor DarkYellow
$response = Invoke-MgGraphRequest -Method Get -Uri $nextLink -ContentType 'application/json' -OutputType HttpResponseMessage
Write-Host "[INFO] Request sent successfully. Response received." -ForegroundColor Green
# Step 8: Convert the response content to a JSON string and parse it into a PSObject
$bytes = $response.Content.ReadAsByteArrayAsync().Result;
$jsonResult = [System.Text.Encoding]::UTF8.GetString($bytes);
Write-Host "[DEBUG] Response content: $jsonResult" -ForegroundColor DarkYellow
$result += $jsonResult | ConvertFrom-Json
# Step 9: Get the next link from the response
$nextLink = $result.'@odata.nextlink'
# Step 10: Dispose of the initial response and perform garbage collection to free up memory
Write-Host "[MEMORY] Disposing initial response and then collection garbage. Trash man.: $nextLink" -ForegroundColor Green
$response.Content.Dispose()
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()
}
# Step 11: Check if there is a skiptoken in the response
if ($result.'@odata.nextlink' -match "skiptoken=([^&]*)" -and $Matches.Count -gt 1) {
$skiptoken = $Matches[1]
Write-Host "[INFO] Found skiptoken: $skiptoken" -ForegroundColor Green
Write-Host "[DEBUG] Skiptoken: $skiptoken" -ForegroundColor DarkYellow
$result | Add-Member -NotePropertyName 'skiptoken' -NotePropertyValue $skiptoken
}
# Step 12: Return the final result
Write-Host "[INFO] Returning result with $($result.records.count) records" -ForegroundColor Green
Write-Host "[DEBUG] Result: $result" -ForegroundColor DarkYellow
return $result
}# This function processes the result received from the API call and saves it to a file.
# It takes an optional parameter 'fileName' which specifies the name of the file to be created.
# If no value is provided for 'fileName', the default value 'file' is used.
Function Process-Results([string]$fileName='file')
{
Write-Host "[INFO] Processing result and saving it to file: $fileName.json" -ForegroundColor Green
# Get the response value from the result object
$response = $result.value
# Read the response content as a byte array
$byteArray = $response.Content.ReadAsByteArrayAsync().Result
# Create a memory stream and write the byte array to it
$stream = New-Object System.IO.MemoryStream
$stream.Write($byteArray, 0, $byteArray.Length)
$stream.Position = 0
# Set the file name and create the file
$file=$fileName + '.json'
Write-Host "[DEBUG] Creating file: $file" -ForegroundColor DarkYellow
[System.IO.StreamWriter]::new($file, $false, [System.Text.Encoding]::UTF8).BaseStream.Write($stream.ToArray(), 0, $stream.Length)
# Create a stream writer to write the response content to the file
$stream = New-Object System.IO.StreamWriter($file, $false, [System.Text.Encoding]::UTF8)
$stream.AutoFlush = $true
# Read the response content as a string and write each line to the file
$response.Content.ReadAsStringAsync().Result | ForEach-Object {
Write-Host "[DEBUG] Writing line: $_" -ForegroundColor DarkYellow
$stream.WriteLine($_)
}
# Flush and close the stream writer
$stream.Flush()
$stream.Close()
# Open the file for reading
$fileStream = [System.IO.File]::OpenRead($file)
# Create a stream reader and JSON text reader to read the file content
$jsonReader = [System.IO.StreamReader]::new($fileStream)
$jsonTextReader = [Newtonsoft.Json.JsonTextReader]::new($jsonReader)
# Create a JSON serializer to deserialize the JSON content
$jsonSerializer = [Newtonsoft.Json.JsonSerializer]::Create()
# Deserialize the JSON content and store it in the 'data' variable
$data = $jsonSerializer.Deserialize($jsonTextReader)
# Close the JSON text reader and stream reader
$jsonTextReader.Close()
$jsonReader.Close()
# Close the file stream
$fileStream.Close()
# Write the deserialized data to the file in JSON format
Write-Host "[DEBUG] Writing deserialized data to file: $file" -ForegroundColor DarkYellow
[System.IO.File]::WriteAllText($file, ($data | ConvertTo-Json -Depth 15))
}The responses are huge JSON strings where I am trying to focus on continuation using skiptoken like below:
{"@odata.context":"https://graph.microsoft.com/beta/$metadata#security/auditLog/queries('de312d1b-b355-4d1a-8124-dffd58352829')/records","@odata.count":150,"@odata.nextLink":"https://graph.microsoft.com/beta/security/auditLog/queries/de312d1b-b355-4d1a-8124-dffd58352829/records?$skiptoken=1!4!MTE-%2f1!48!NGY2ZjlhYmEtNDgxZC00NGViLTgzOTctOGU3NTFmMGUzNzgx","value":[{ //1099440 length response; // with request header Accept-Encoding: gzip 792412 }]this is just how I am approaching it though.
Chunked splitting / processing should do the trick because it is similar to the batching for Search-UnifiedAuditLog
For this to be effective, batching with the Exchange API permissions are required. It offloads the powershell processing to the server-side but is required to be split in chunks similar to below.
function New-ExoBulkRequest ($tenantid, $cmdletArray, $useSystemMailbox, $Anchor, $NoAuthCheck, $Select)
{
<#
.FUNCTIONALITY
This is not from the omdule this is a sample of custom function.
#>
if ((Get-AuthorisedRequest -TenantID $tenantid) -or $NoAuthCheck -eq $True)
{
$token = Get-Token -resource 'https://outlook.office365.com' -Tenantid $tenantid
$Tenant = Get-Tenants -IncludeErrors | Where-Object { $_.defaultDomainName -eq $tenantid -or $_.customerId -eq $tenantid }
$Headers = @{
Authorization = "Bearer $($token.access_token)"
Prefer = 'odata.maxpagesize = 1000;odata.continue-on-error'
'parameter-based-routing' = $true
'X-AnchorMailbox' = $Anchor
}
try
{
if ($Select) { $Select = "`$select=$Select" }
$URL = "https://outlook.office365.com/adminapi/beta/$($tenant.customerId)/InvokeCommand?$Select"
$BatchURL = "https://outlook.office365.com/adminapi/beta/$($tenant.customerId)/`$batch"
$BatchBodyObj = @{
requests = @()
}
# Split the cmdletArray into batches of 10
$batches = [System.Collections.ArrayList]@()
for ($i = 0; $i -lt $cmdletArray.Length; $i += 10)
{
$null = $batches.Add($cmdletArray[$i..[math]::Min($i + 9, $cmdletArray.Length - 1)])
}
# Process each batch
$ReturnedData = foreach ($batch in $batches)
{
$BatchBodyObj.requests = [System.Collections.ArrayList]@()
foreach ($cmd in $batch)
{
$cmdparams = $cmd.CmdletInput.Parameters
if ($cmdparams.Identity) { $Anchor = $cmdparams.Identity }
if ($cmdparams.anr) { $Anchor = $cmdparams.anr }
if ($cmdparams.User) { $Anchor = $cmdparams.User }
if (!$Anchor -or $useSystemMailbox)
{
$OnMicrosoft = $Tenant.initialDomainName
$Anchor = "UPN:SystemMailbox{8cc370d3-822a-4ab8-a926-bb94bd0641a9}@$($OnMicrosoft)"
}
$headers['X-AnchorMailbox'] = $Anchor
$Headers['X-CmdletName'] = $cmd.CmdletInput.CmdletName
$headers['Accept'] = 'application/json; odata.metadata=minimal'
$headers['Accept-Encoding'] = 'gzip'
$BatchRequest = @{
url = $URL
method = 'POST'
body = $cmd
headers = $Headers.Clone()
id = "$(New-Guid)"
}
$null = $BatchBodyObj['requests'].add($BatchRequest)
}
$Results = Invoke-RestMethod $BatchURL -ResponseHeadersVariable responseHeaders -Method POST -Body (ConvertTo-Json -InputObject $BatchBodyObj -Depth 10) -Headers $Headers -ContentType 'application/json; charset=utf-8'
$Results
Write-Host "Batch #$($batches.IndexOf($batch) + 1) of $($batches.Count) processed"
}
}
catch
{
$ErrorMess = $($_.Exception.Message)
$ReportedError = ($_.ErrorDetails | ConvertFrom-Json -ErrorAction SilentlyContinue)
$Message = if ($ReportedError.error.details.message)
{
$ReportedError.error.details.message
}
elseif ($ReportedError.error.message) { $ReportedError.error.message }
else { $ReportedError.error.innererror.internalException.message }
if ($null -eq $Message) { $Message = $ErrorMess }
throw $Message
}
$FinalData = foreach ($item in $ReturnedData.responses.body)
{
if ($item.'@adminapi.warnings')
{
Write-Warning $($item.'@adminapi.warnings' | Out-String)
}
if ($item.error)
{
if ($item.error.details.message)
{
$msg = [pscustomobject]@{error = $item.error.details.message; target = $item.error.details.target }
}
else
{
$msg = [pscustomobject]@{error = $item.error.message; target = $item.error.details.target }
}
$item | Add-Member -MemberType NoteProperty -Name 'value' -Value $msg -Force
}
[pscustomobject]$item.value
}
return $FinalData
}
else
{
Write-Error 'Not allowed. You cannot manage your own tenant or tenants not under your scope'
}
}
this abovesnippet is for batching data with large result sets. It splits the $cmdletArray into batches of a specified size (in this case, 10) and then processes each batch.
Here's the batching process:
The code first checks if the user is authorized to make a request to the specified tenant.
If the user is authorized.
Gets tenantinfo
sets headers
checks if the $Select parameter is provided
It splits the $cmdletArray into batches of size
foreach batch createse request and sets body and headers
executes api call
It processes the data thenreturns the final data.
The batching process helps to handle large result sets by splitting them into smaller chunks and processing them in batches. This can help avoid memory issues and improve performance.
Nice references internally for why the ExchangeOnlineManagement module is a shitshow. Literally every other module these days that is from msoft and still active is written binary as C# for powershell except for exchangeonline, it would be titan of a job though.
# .ExternalHelp Microsoft.Exchange.RecordsandEdge-Help.xml
function script:Search-UnifiedAuditLog
{
[CmdletBinding(DefaultParameterSetName='Identity')]
param(
[Parameter(ParameterSetName='Identity', Mandatory=$true)]
${EndDate},
[Parameter(ParameterSetName='Identity')]
[switch]
${Formatted},
[Parameter(ParameterSetName='Identity')]
[string]
${FreeText},
[Parameter(ParameterSetName='Identity')]
[switch]
${HighCompleteness},
[Parameter(ParameterSetName='Identity')]
[string[]]
${IPAddresses},
[Parameter(ParameterSetName='Identity')]
[string]
${LongerRetentionEnabled},
[Parameter(ParameterSetName='Identity')]
[string[]]
${ObjectIds},
[Parameter(ParameterSetName='Identity')]
[string[]]
${Operations},
[Parameter(ParameterSetName='Identity')]
${RecordType},
[Parameter(ParameterSetName='Identity')]
[ValidateRange(1, 5000)]
[int]
${ResultSize},
[Parameter(ParameterSetName='Identity')]
${SessionCommand},
[Parameter(ParameterSetName='Identity')]
[string]
${SessionId},
[Parameter(ParameterSetName='Identity')]
[string[]]
${SiteIds},
[Parameter(ParameterSetName='Identity', Mandatory=$true)]
${StartDate},
[Parameter(ParameterSetName='Identity')]
[string[]]
${UserIds}
)
Begin {
$CmdletRequestId, $cmdletRequestIdGeneratedInBegin, $cmdletIDList = Init-CmdletBegin -CmdletParameters $MyInvocation.BoundParameters -CmdletName $MyInvocation.MyCommand.Name
try
{
$UseBatching, $BatchBodyObj, $BatchRequestParameters = Execute-CmdletBatchingBeginBody -EnableBatchingStatus:$true -ExpectingInput $PSCmdlet.MyInvocation.ExpectingInput
}
catch
{
Commit-CmdletLogOnError -CmdletRequestId $CmdletRequestId -ErrorRecord $_
$global:EXO_LastExecutionStatus = $false;
throw $_
}
}
Process {
$cmdletRequestIdGeneratedInBegin, $cmdletRequestIdGeneratedInProcess, $CmdletRequestId, $cmdletIDList = Init-CmdletProcess -CmdletRequestIdGeneratedInBegin $cmdletRequestIdGeneratedInBegin -UseBatching $UseBatching -BatchBodyObj $BatchBodyObj -CmdletRequestId $CmdletRequestId -CmdletIDList $cmdletIDList -CmdletParameters $MyInvocation.BoundParameters -CmdletName $MyInvocation.MyCommand.Name
try
{
Execute-Command -CmdletName 'Search-UnifiedAuditLog' -Parameters $PSBoundParameters -CmdletRequestId $CmdletRequestId
}
catch
{
Commit-CmdletLogOnError -CmdletRequestId $CmdletRequestId -ErrorRecord $_
$global:EXO_LastExecutionStatus = $false;
throw $_
}
finally
{
Log-EndTimeInCmdletProcessBlock -UseBatching $UseBatching -CmdletRequestId $CmdletRequestId
}
}
End {
Execute-CmdletEndBlock -BatchBodyObj $BatchBodyObj -CmdletRequestId $CmdletRequestId -CmdletIDList $cmdletIDList
}
}function Execute-Command
{
[CmdletBinding()]
param(
[Parameter(Mandatory)]
[string]
$CmdletName,
[Parameter(Mandatory)]
$Parameters,
$ParameterBasedRoutingHintDelegate,
[string]
$ParameterBasedRoutingHintParameterValue,
[boolean]
$ShowConfirmPrompt = $false,
$PSCmdletObject,
[boolean]
$UseParameterBasedRouting = $false,
[string]
$CmdletRequestId
)
process
{
$script:DefaultPageSize = 1000
if ($ConnectionContextObjectId -ne $null)
{
$ConnectionContext = [Microsoft.Exchange.Management.ExoPowershellSnapin.ConnectionContextFactory]::GetCurrentConnectionContext($ConnectionContextObjectId)
if (($ConnectionContext -ne $null) -and ($ConnectionContext.PageSize -ge 1) -and ($ConnectionContext.PageSize -le 1000))
{
$script:DefaultPageSize = $ConnectionContext.PageSize
}
}
$resultSize = $script:DefaultPageSize
$nextPageSize = $script:DefaultPageSize
$anotherPagedQuery = $true
# Set the generic headers
$Headers = @{
'Accept' = 'application/json'
'Accept-Charset' = 'UTF-8'
'Content-Type' = 'application/json'
}
# Set cmdlet name in request header
$Headers['X-CmdletName'] = $CmdletName
# Set a client requestid to uniquely identify this cmdlet execution on the server.
# This id will be the same for all retries, and all requests in a batch so it is set here itself.
$Headers['client-request-id'] = $CmdletRequestId
$WarningActionValue = ''
if ($Parameters.ContainsKey('WarningAction'))
{
$WarningActionValue = $Parameters['WarningAction']
if($warningActionValue -eq 'Ignore')
{
# Treating Ignore WarningAction as SilentlyContinue to maintain RPS parity
# This is for PS understanding to treat Ignore as SilentlyContinue
$WarningPreference = 'SilentlyContinue'
# This is for sending to server side
$WarningActionValue = 'SilentlyContinue'
}
}
$Headers['WarningAction'] = $WarningActionValue
$null = $Parameters.Remove('UseCustomRouting');
# Construct the request body
$params = @{}
$cmdletClass = Get-Command $CmdletName
$params = Transform-Parameters -Parameters $Parameters -cmdletClass $cmdletClass
if ($ShowConfirmPrompt)
{
if ($params.ContainsKey('Confirm') -eq $false)
{
$params.Add('Confirm', $true)
}
$UseBatching = $false
}
$cmdletInput = @{}
$cmdletInput['CmdletName'] = $CmdletName
$cmdletInput['Parameters'] = $params
$BodyObj = @{}
$BodyObj['CmdletInput'] = $cmdletInput
if ($UseBatching)
{
AddTo-BatchRequest -BodyObj $BodyObj -ParameterBasedRoutingHintDelegate $ParameterBasedRoutingHintDelegate -ParameterBasedRoutingHintParameterValue $ParameterBasedRoutingHintParameterValue -UseParameterBasedRouting $UseParameterBasedRouting -CmdletRequestId $CmdletRequestId
if($BatchBodyObj['requests'].Count -eq 10)
{
$BatchResponse = Execute-BatchRequest -CmdletRequestId $CmdletRequestId
Deserialize-BatchRequest -BatchResponse $BatchResponse -CmdletRequestId $CmdletRequestId
$BatchBodyObj['requests'] = New-Object System.Collections.ArrayList;
$endTime = Get-Date;
Log-Column -ColumnName EndTime -CmdletId $CmdletRequestId -Value $endTime
}
return
}
$Body = ConvertTo-Json $BodyObj -Depth 10 -Compress
$Body = ([System.Text.Encoding]::UTF8.GetBytes($Body));
$resultSize = Get-ResultSize $Parameters
if ($resultSize -eq -1)
{
return
}
if ($resultSize -ne "Unlimited")
{
$nextPageSize = [Math]::min($resultSize, $script:DefaultPageSize)
}
$nextPageUri = ''
while ($anotherPagedQuery)
{
# Update the next page size in the header
$Headers['Prefer'] = "odata.maxpagesize={0}"-f $nextPageSize
$waitTime = 0 # In milliseconds
$isRetryHappening = $true
$isQuerySuccessful = $false
$claims = [string]::Empty
for ($retryCount = 0; $retryCount -le $script:DefaultMaxRetryTimes -and $isRetryHappening; $retryCount++)
{
try
{
if ($ConnectionContextObjectId -ne $null)
{
$ConnectionContext = [Microsoft.Exchange.Management.ExoPowershellSnapin.ConnectionContextFactory]::GetCurrentConnectionContext($ConnectionContextObjectId);
$AuthInfo = $ConnectionContext.GetAuthHeader($claims, $cmdletRequestId);
$claims = [string]::Empty
}
else
{
# ConnectionContextObjectId is not present, so request is from an older version of client. Hence use AuthHeaderUtils.
$AuthInfo = Get-AuthInfoUsingTokenProvider $TokenProviderObjectId
}
# Initialize per request logging
$startTime = Get-Date
Init-Log -CmdletId $CmdletRequestId
Log-Column -ColumnName StartTime -CmdletId $CmdletRequestId -Value $startTime
Log-Column -ColumnName CmdletName -CmdletId $CmdletRequestId -Value $CmdletName
Log-Column -ColumnName CmdletParameters -CmdletId $CmdletRequestId -Value $Parameters
$Uri = if ([string]::IsNullOrWhitespace($nextPageUri)) {$AuthInfo.BaseUri + '/' + $AuthInfo.TenantId + '/InvokeCommand'} else {$nextPageUri};
Populate-Headers -ParameterBasedRoutingHintDelegate $ParameterBasedRoutingHintDelegate -ParameterBasedRoutingHintParameterValue $ParameterBasedRoutingHintParameterValue -UseParameterBasedRouting $UseParameterBasedRouting -AuthInfo $AuthInfo -Headers $Headers
$existingProgressPreference = $global:ProgressPreference
$global:ProgressPreference = 'SilentlyContinue'
if ($ShowConfirmPrompt)
{
Write-Verbose('Validation call to the server:')
}
$timeout = Get-HttpTimeout $Headers
$Result = Invoke-WebRequest -UseBasicParsing -TimeoutSec $timeout -Method 'POST' -Headers $Headers -Body $Body -Uri $Uri
$confirmationMessage = ''
if ($Result -ne $null -and $Result.Content -ne $null)
{
$ResultObject = (ConvertFrom-JSON $Result.Content)
$confirmationMessage = $ResultObject.'@adminapi.confirmationmessage';
}
#ShouldProcess call should be made before checking the value of $ShowConfirmPrompt for supporting WhatIf behavior
if ($PSCmdletObject -ne $null -and $PSCmdletObject.ShouldProcess($confirmationMessage, $null, $null) -and $ShowConfirmPrompt)
{
$params['Confirm'] = $false
$cmdletInput = @{}
$cmdletInput['CmdletName'] = $CmdletName
$cmdletInput['Parameters'] = $params
$BodyObj = @{}
$BodyObj['CmdletInput'] = $cmdletInput
$Body = ConvertTo-Json $BodyObj -Depth 10 -Compress
$Body = ([System.Text.Encoding]::UTF8.GetBytes($Body));
Write-Verbose('Execution call to the server:')
$timeout = Get-HttpTimeout $Headers
$Result = Invoke-WebRequest -UseBasicParsing -TimeoutSec $timeout -Method 'POST' -Headers $Headers -Body $Body -Uri $Uri
}
$global:ProgressPreference = $existingProgressPreference
$isQuerySuccessful = $true
$isRetryHappening = $false
$endTime = Get-Date;
Log-Column -ColumnName EndTime -CmdletId $CmdletRequestId -Value $endTime
foreach ($id in $cmdletIDList)
{
Commit-Log -CmdletId $id
}
}
catch
{
$global:ProgressPreference = $existingProgressPreference
$claims = Get-ClaimsFromExceptionDetails -ErrorObject $_
$isRetryable = CheckRetryAndHandleWaitTime -retryCount $retryCount -ErrorObject $_ -CmdletRequestId $CmdletRequestId -IsFromBatchingRequest $false
if ($isRetryable -eq $false)
{
$global:EXO_LastExecutionStatus = $false;
return
}
else
{
# Log error for per request logging if it is retriable error, non retriable error gets logged in CheckRetryAndHandleWaitTime
Log-Column -ColumnName GenericError -CmdletId $CmdletRequestId -Value $_
$endTime = Get-Date;
Log-Column -ColumnName EndTime -CmdletId $CmdletRequestId -Value $endTime
foreach ($id in $cmdletIDList)
{
Commit-Log -CmdletId $id
}
}
}
}
# Result Handling With Pagination Response
if ($Result -ne $null -and $Result.Content -ne $null)
{
$ResultObject = (ConvertFrom-JSON $Result.Content)
# Print the Result
PrintResultAndCheckForNextPage -ResultObject $ResultObject -VerifyNextPageLinkCheckRequired $true -anotherPagedQuery ([ref]$anotherPagedQuery) -resultSize $resultSize -nextPageSize $nextPageSize -isFromBatchingRequest:$false
if ($anotherPagedQuery -eq $true)
{
$nextPageUri = $ResultObject.'@odata.nextLink'
if ($resultSize -ne 'Unlimited')
{
$resultSize -= $nextPageSize
$nextPageSize = [Math]::min($resultSize, $script:DefaultPageSize)
}
}
}
else
{
$anotherPagedQuery = $false
}
}
}
}
function AddTo-BatchRequest
{
[CmdletBinding()]
param(
[Parameter(Mandatory)]
$BodyObj,
$ParameterBasedRoutingHintDelegate,
[string]
$ParameterBasedRoutingHintParameterValue,
[boolean]
$UseParameterBasedRouting = $false,
[string]
$CmdletRequestId
)
process
{
$resultSize = Get-ResultSize $Parameters
if ($resultSize -ne "Unlimited" -and $resultSize -gt 0)
{
$nextPageSize = [Math]::min($resultSize, $script:DefaultPageSize)
}
$claims = [string]::Empty
if ($ConnectionContextObjectId -ne $null)
{
$ConnectionContext = [Microsoft.Exchange.Management.ExoPowershellSnapin.ConnectionContextFactory]::GetCurrentConnectionContext($ConnectionContextObjectId);
$AuthInfo = $ConnectionContext.GetAuthHeader($claims, $cmdletRequestId);
$claims = [string]::Empty
}
else
{
# ConnectionContextObjectId is not present, so request is from an older version of client. Hence use AuthHeaderUtils.
$AuthInfo = Get-AuthInfoUsingTokenProvider $TokenProviderObjectId
}
$Uri = $AuthInfo.BaseUri + '/' + $AuthInfo.TenantId + '/InvokeCommand';
Populate-Headers -ParameterBasedRoutingHintDelegate $ParameterBasedRoutingHintDelegate -ParameterBasedRoutingHintParameterValue $ParameterBasedRoutingHintParameterValue -UseParameterBasedRouting $UseParameterBasedRouting -AuthInfo $AuthInfo -Headers $Headers
# Update the next page size along with continue on error preference in the header
$Headers['Prefer'] = "odata.maxpagesize={0};odata.continue-on-error"-f $nextPageSize
Write-Verbose('Adding {0} Request to Batch' -f ($BatchBodyObj['requests'].Count + 1))
$BatchRequest = @{}
$BatchRequest['url'] = $Uri
$BatchRequest['method'] = 'POST'
$BatchRequest['body'] = $BodyObj
$BatchRequest['headers'] = $Headers
$BatchRequest['id'] = $BatchBodyObj['requests'].Count.ToString()
$null = $BatchBodyObj['requests'].add($BatchRequest);
$null = $BatchRequestParameters.add($Parameters);
}
}
function Deserialize-BatchRequest
{
[CmdletBinding()]
param(
[Parameter(Mandatory)]
$BatchResponse,
[string]
$CmdletRequestId
)
process
{
# Variables used to handle next paged requests
$resultSize = $script:DefaultPageSize
$nextPageSize = $script:DefaultPageSize
$anotherPagedQuery = $true
if ($BatchResponse -eq $null -or $BatchResponse.Content -eq $null)
{
return
}
$BatchBody = ConvertTo-Json $BatchBodyObj -Depth 10
$BatchBodyPSObject = ConvertFrom-Json -InputObject $BatchBody
$batchBodyIterator = 0
$BatchedResultObject = (ConvertFrom-JSON $BatchResponse.Content)
ForEach ($Result in $BatchedResultObject.responses)
{
$Result.headers | Add-Member -MemberType NoteProperty -Name 'Cmdlet-Name' -Value $BatchBodyPSObject.requests[$batchBodyIterator++].body.CmdletInput.CmdletName
if ($Result.status -eq 200)
{
$ResultObject = $Result.body
$RequestID = [int]$Result.id
$CurrentRequestParameters = $BatchRequestParameters[$RequestID]
$resultSize = Get-ResultSize $CurrentRequestParameters
if ($resultSize -ne "Unlimited" -and $resultSize -gt 0)
{
$nextPageSize = [Math]::min($resultSize, $script:DefaultPageSize)
}
# Print Result
PrintResultAndCheckForNextPage -ResultObject $ResultObject -VerifyNextPageLinkCheckRequired $true -anotherPagedQuery ([ref]$anotherPagedQuery) -resultSize $resultSize -nextPageSize $nextPageSize
if ($anotherPagedQuery)
{
$PagedRequest = $BatchBodyObj['requests'][$RequestID]
Execute-BatchedNextPageRequest -PagedRequest $PagedRequest -PagedResponse $ResultObject -ResultSize $resultSize -NextPageSize $nextPageSize -CmdletRequestId $CmdletRequestId
}
}
else
{
#Retry individual request
$RequestID = [int]$Result.id
$FailedRequest = $BatchBodyObj['requests'][$RequestID]
$CustomException = Create-CustomError $Result $BatchResponse.Headers
Retry-SingleBatchedRequest -FailedRequest $FailedRequest -ErrorObject $CustomException -CmdletRequestId $CmdletRequestId
}
}
}
}Thanks again, @Calvindd2f, for the detailed explanation. I need to read this more carefully when I have some free time to wrap my head around it and understand what is going on. Haha.
JoeyInvictus
commented
3 months ago @Calvindd2f, code snippeds look good. I’m going to need some time to test, understand, and process this for the Extractor Suite (currently quite busy).
@angry-bender have you had a chance to test this to see if it fixes your memory issues? I currently only have access to a small tenant, so I'm unable to test if it would fix the issues.
I can try to implement this, but I won't be able to confirm if it fixes the issue. Based on the code, it seems promising, but I can't be certain without further testing. Need a new BEC case with a big tenant to play around with this ;P
angry-bender
commented
3 months ago @Calvindd2f, code snippeds look good. I’m going to need some time to test, understand, and process this for the Extractor Suite (currently quite busy).
@angry-bender have you had a chance to test this to see if it fixes your memory issues? I currently only have access to a small tenant, so I'm unable to test if it would fix the issues.
I can try to implement this, but I won't be able to confirm if it fixes the issue. Based on the code, it seems promising, but I can't be certain without further testing. Need a new BEC case with a big tenant to play around with this ;P
Unfortunately not yet, but I can try to test it with a larger tenant Tommorow. Been quite busy here too unfortunately
angry-bender
commented
3 months ago @Calvindd2f, code snippeds look good. I’m going to need some time to test, understand, and process this for the Extractor Suite (currently quite busy).
@angry-bender have you had a chance to test this to see if it fixes your memory issues? I currently only have access to a small tenant, so I'm unable to test if it would fix the issues.
I can try to implement this, but I won't be able to confirm if it fixes the issue. Based on the code, it seems promising, but I can't be certain without further testing. Need a new BEC case with a big tenant to play around with this ;P
Unfortunately not yet, but I can try to test it with a larger tenant Tommorow. Been quite busy here too unfortunately
Running a test in this one now 😊. I've manually copied and pasted the above however @Calvindd2f would you mind giving us a PR, just to make sure I have all the code right?
angry-bender
commented
3 months ago @Calvindd2f, code snippeds look good. I’m going to need some time to test, understand, and process this for the Extractor Suite (currently quite busy).
@angry-bender have you had a chance to test this to see if it fixes your memory issues? I currently only have access to a small tenant, so I'm unable to test if it would fix the issues.
I can try to implement this, but I won't be able to confirm if it fixes the issue. Based on the code, it seems promising, but I can't be certain without further testing. Need a new BEC case with a big tenant to play around with this ;P
Unfortunately not yet, but I can try to test it with a larger tenant Tommorow. Been quite busy here too unfortunately
Running a test in this one now 😊. I've manually copied and pasted the above however @Calvindd2f would you mind giving us a PR, just to make sure I have all the code right?
Just to confirm here, a PR would help, I am getting some exceptions for calling .cstor under the streamWriter line and the term end before the line #region results
JoeyInvictus
commented
3 months ago Hi,
We have a new BEC incident, and last night I was experiencing the same issues you described. I tried using the Get-MgBetaSecurityAuditLogQueryRecord cmdlet, but it resulted in the same error. I did some Googling and found some GitHub issues on the Microsoft GitHub page. So, I think it's a Microsoft backend issue and not an Extractor issue.
https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2855
Calvindd2f
commented
3 months ago Sorry I've been a bit caught up in some other personal project for a while so kinda tunnel visioned on that.
I'll try spin up a POC but with the last comment left by Joey , I will try be wary. I suppose the main focus is that when the bytestream is converted to string , it isn't converted too output with errors.
Out of curiosity I've opened up the DLL which contains Get-MgBetaSecurityAuditLogQueryRecord in dotPeek and checked the method where the error is thrown. There is nothing immediately wrong / off so we might be at the mercy of Microsoft as to why the output is getting escaped/throwing terminating error.
JoeyInvictus
commented
3 months ago @Calvindd2f Thank you! I'm curious to see your findings. I suspect it might be an issue with the MS backend, but I believe you'll be able to figure this out better than I can.
JoeyInvictus
commented
2 months ago Looks like there are some issues at Microsoft: https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2927
I have a support ticket but no updates so far.
JoeyInvictus
commented
3 weeks ago Not sure what is going on with the Graph module, but it seems like they removed the Get-MgBetaSecurityAuditLogQuery command as well. The documentation gives a 404 error code:
https://learn.microsoft.com/en-us/powershell/module/microsoft.graph.beta.security/get-mgbetasecurityauditlogquery?view=graph-powershell-beta
When looking at this documentation: https://learn.microsoft.com/en-us/graph/api/security-auditlogquery-get?view=graph-rest-beta&tabs=python
They removed the PowerShell example. When you check the Create AuditLogQuery, it's still there, and I can still start the scan with New-MgBetaSecurityAuditLogQuery. Wondering what is going on around the Graph UAL haha. Still waiting on my support ticket for almost 2 months as well...
JoeyInvictus
commented
1 week ago I finally had a chat with Microsoft, and they advised that we shouldn't use the BETA features for our tooling, as they can't provide support for them... Since it's not an issue with this tool specifically, I will close the issue from now. Unfortunately not much I can do at this point.
When using
Get-UALAll, the search has completed, however for UAL with >400,000 records (>1GB in size) it looks like it is erroring out with the following exception:Conversion from JSON failed with error: After parsing a value an unexpected character was encountered: {. Path 'value[9].administrativeUnits', line 1, position 19182.Additionally, it took > 1 hour this point for a fairly small tenancy (compared to the last version), where i put in some extra messaging in #74 to aid in knowing whats happening in the process
I'm not sure on a workaround for this one, unless there is a way to split the amount of results by a certain size when requesting back from the API