Azure Functions PS + EXO V3 -- Possible memory leak?

iyerusad commented 1 year ago

Looks somewhat similar to #701 , including reproducing only after some time (e.g. possible memory leak):

Initially posted in EXO V3 github (https://github.com/MicrosoftDocs/office-docs-powershell/issues/11264) as seemed to concretely happen after migrating from powershell module EXO v2 to EXO v3.

However reading #701 sounded very similar and posting here as this might be a Azure Functions Powershell runtime issue, but not sure. Perhaps this error messages provide additional insight.

Thought: Is EXO V3 PS toppling an Azure Functions Powershell limit (1.5GB RAM)?

Error messages (in Azure Function logs):

WARNING: Connecting to ExchangeOnline -> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired) at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired) at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote) at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonTextReader.Read() at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) at Newtonsoft.Json.Linq.JTokenWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken(JsonReader reader, JsonContract contract) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at Microsoft.Exchange.Management.ExoPowershellSnapin.NewEXOModule.ProcessRecord() at System.Management.Automation.Cmdlet.DoProcessRecord() at System.Management.Automation.CommandProcessor.ProcessRecord()

Result: Failure Exception: Connecting to ExchangeOnline -> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired) at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired) at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote) at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonTextReader.Read() at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) at Newtonsoft.Json.Linq.JTokenWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken(JsonReader reader, JsonContract contract) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at Microsoft.Exchange.Management.ExoPowershellSnapin.NewEXOModule.ProcessRecord() at System.Management.Automation.Cmdlet.DoProcessRecord() at System.Management.Automation.CommandProcessor.ProcessRecord() Stack: at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input) at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke) at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync) at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection1 input, PSDataCollection1 output, PSInvocationSettings settings) at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection1 input, PSDataCollection1 output, PSInvocationSettings settings) at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection1 output, PSInvocationSettings settings) at System.Management.Automation.PowerShell.Invoke[T](IEnumerable input, IList1 output, PSInvocationSettings settings) at System.Management.Automation.PowerShell.Invoke[T]() at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellExtensions.InvokeAndClearCommands[T](PowerShell pwsh) in D:\a_work\1\s\src\PowerShell\PowerShellExtensions.cs:line 45 at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.InvokeNonOrchestrationFunction(DurableController durableController, IDictionary outputBindings) in D:\a_work\1\s\src\PowerShell\PowerShellManager.cs:line 301 at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.InvokeFunction(AzFunctionInfo functionInfo, Hashtable triggerMetadata, TraceContext traceContext, RetryContext retryContext, IList`1 inputData, FunctionInvocationPerformanceStopwatch stopwatch) in D:\a_work\1\s\src\PowerShell\PowerShellManager.cs:line 230 at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.InvokeFunction(AzFunctionInfo functionInfo, PowerShellManager psManager, FunctionInvocationPerformanceStopwatch stopwatch, InvocationRequest invocationRequest) in D:\a_work\1\s\src\RequestProcessor.cs:line 336 at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessInvocationRequestImpl(StreamingMessage request, AzFunctionInfo functionInfo, PowerShellManager psManager, FunctionInvocationPerformanceStopwatch stopwatch) in D:\a_work\1\s\src\RequestProcessor.cs:line 308

Azure Function: `msExch-Get-MailBox`

using namespace System.Net
param($Request)
$funcName = "msExch-Get-MailBox"

##########################
#   Starting Function:   #
##########################
Write-Information "Started Function: $funcName"

function errorHandler{
  #Version: errorHandlerPS-0.1
  param (
    [Parameter(Position = 0, Mandatory=$true)][Object[]]$errorObj,
    [Parameter(Position = 1, Mandatory=$false)][string]$prefix,
    [Parameter(Position = 2, Mandatory=$false)][switch][Alias("fatal")]$exception
  )

  $errorMsg = "$errorObj"
  if ($prefix) {$errorMsg = "$prefix $errorMsg"}
  if ($exception) {$exceptions.Add($errorMsg) | Out-Null}
  else {$errors.Add($errorMsg) | Out-Null}
  Write-Warning $errorMsg
}
function noErrors{
  #Version: noErrorsPS-0.1
  if ($errors.count -eq 0 -and $exceptions.count -eq 0) {return $true}
  else {return $false}
}

$errors = [System.Collections.ArrayList]@()
$exceptions = [System.Collections.ArrayList]@()
$appRegistrationID = "**snipped**"
$tenantDomain = "**snipped**.onmicrosoft.com"
$certName = "ExchangeAZFuncAuth";
$isConnected = $false;
$mailboxes = $null;
$result = $null;

# Locate App Registration and Certificate
$cert = Get-ChildItem -Path "Cert:/**/MY" -Recurse | Where-Object {$_.Subject -Match "$certName"};
if ($cert) {Write-Host "Located auth cert, thumbprint: $($cert.thumbprint)"}
else {errorHandler -err "FATAL: Was unable to find authentication certificate (name: '$certName')" -fatal}

# ExchangeOnline: Connect to session
if (noErrors -and $cert -and $cert.thumbprint) {
  Write-Information "Importing Module: ExchangeOnlineManagement"
  Import-Module ExchangeOnlineManagement

  if (([bool](Get-Command -Name "Connect-ExchangeOnline" -ErrorAction SilentlyContinue))) {
    try {
      Write-Information "ExchangeOnline: Connecting..."
      Connect-ExchangeOnline -Certificate $cert -AppId $appRegistrationID -Organization $tenantDomain -Showbanner:$false -EA Stop

      $isConnected = Get-AcceptedDomain

      if ($isConnected) {Write-Information " -- Connected to ExchangeOnline session."}
      else {throw " -- Failed to connect."}
    }
    catch {errorHandler -err $_.Exception -pre "Connecting to ExchangeOnline ->" -fatal}
  } else {errorHandler -err "Couldn't find command (Connect-ExchangeOnline), is the module loaded??" -fatal}
}

# ExchangeOnline: Get mailboxes
if (noErrors -and $session) {
  if (([bool](Get-Command -Name "Get-Mailbox" -ErrorAction SilentlyContinue))) {
    Write-Information "ExchangeOnline: Getting mailboxes..."
    $mailboxes = Get-Mailbox
    Write-Information " -- Finished getting mailboxes."
  } else{errorHandler -err "Couldn't find command (Get-Mailbox), is the module loaded??"}
}

# ExchangeOnline: Close the session
if ($isConnected) {
  try {
    Write-Information "ExchangeOnline: Closing session..."
    Disconnect-ExchangeOnline -Confirm:$false
    Write-Information " -- Closed."
  }
  catch {
    errorHandler -err $_.Exception -pre "ExchangeOnline: Failed to close session! ->"
  }
}

# Define the result
if ($mailboxes) {
  $result = @{
    mailboxes = $mailboxes
  }
}

######################
# Wrap up the function
######################
if ($result) {
  $httpResponse = [HttpResponseContext]@{
    StatusCode = 200
    Body = $result
  }
} elseif ((!$result) -and (noErrors)) {
  errorHandler("Unknown failure: No errors logged but function did NOT complete as expected")
}

if ($exceptions.count -gt 0) {
  Write-Error "Exit: 500; Encountered fatal error"
  $httpResponse = [HttpResponseContext]@{
    StatusCode = 500
    Body = @{
      message = "FATAL failure: server side failure"
      exceptions = $exceptions -join ', '
      errors = $errors -join ', '
    }
  }
} elseif ($errors.count -gt 0) {
  #Not 422 since no inputs, any errors = 500
  Write-Warning "Exit: 500; Encountered errors during execution"
  $httpResponse = [HttpResponseContext]@{
    StatusCode = 500
    Body = @{
      message = "Exit: 500; Encountered errors during execution"
      errors = $errors -join ', '
    }
  }
}
Write-Information "Finished Function: $funcName"

#Return/Apply the HTTP Response
Push-OutputBinding -Name Response -Value $httpResponse
if ($exceptions.count -gt 0 -or $errors.count -gt 0) {
  throw "$($exceptions -join ', ') $($errors -join ', ')" # Must throw exception for AzureFunctions to mark as failed (even 500s)
}

Git commit migrating to EXO v3 (no newer commits):

Francisco-Gamino commented 1 year ago

Hello @iyerusad -- Could you please open a support case via the Azure Portal? We will need to collaborate with the ExchangeOnlineManagement team to help them gather any telemetry they need from our worker VMs, so that they can investigate and resolve this issue. Thank you.

/cc @chrisda @AnatoliB @shreyabatra4

Francisco-Gamino commented 1 year ago

I will go ahead and close this issue.

iyerusad commented 1 year ago

I am unable to create a Support Request via Azure Portal because it only allows me to create support requests for Billing, Subscription Management, and Quota increase... this appears to be a Bug/Technical Issue. In the EXO V3 repo issue I documented some more troubleshooting and it appears to crash like clockwork at 16 runs.

Is there a way to report this likely bug? Azure Portal or elsewhere? I have attempted to report in this repo, in the EXO V3 repo, and both point to Azure Portal who wants me to upgrade our tenant to $29/m to report a Technical issue.

Francisco-Gamino commented 1 year ago

Hello @iyerusad -- Could you please check to see if you have any supports plans under Help + support - Microsoft Azure. If not, you might need to sign up for a Developer Support which start from $29 a month. For this, your tenant admin should be able to help. Thank you.

iyerusad commented 1 year ago

It appears dotnet.exe bloats to fill memory capacity and doesn't release. You can repro this locally (not Azure) by running same code until it eats up all the machine RAM. Just tested locally and with 30 runs dotnet.exe has bloated to 5.194GB and will fill out my devbox's RAM until it fails too or I kill the process.

Azure Functions Core Tools Core Tools Version: 4.0.5198 Commit hash: N/A (64-bit) Function Runtime Version: 4.21.1.20667

Azure / azure-functions-powershell-worker