search

microsoftgraph / msgraph-sdk-powershell

Powershell SDK for Microsoft Graph
https://www.powershellgallery.com/packages/Microsoft.Graph
Other
678 stars 157 forks source link

`Microsoft.Graph.Beta.Identity.Governance` module is too big and fails on Azure Automation #2516

Open jpawlowski opened 6 months ago

jpawlowski commented 6 months ago

When using Microsoft.Graph.Identity.Governance module in Azure Automation runbooks running inside the Azure sandbox, the memory limitation of 400 MB becomes profoundly serious.

I noticed that the actual size of the Microsoft.Graph.Identity.Governance module is already huge (Version 2.11.1 has 17,131.3 KB), and the beta module is even bigger using 24,481.4 KB of disk space. This gives quite a good indication about the memory consumption of this module.

Importing that module inside a PowerShell 5.1 runbook using Import-Module Microsoft.Graph.Beta.Identity.Governance will often fail in Azure Automation, leading to out of memory exceptions:

ForEach-Object : Exception of type 'System.OutOfMemoryException' was thrown. At C:\usr\src\PSModules\Microsoft.Graph.Beta.Identity.Governance\internal\Microsoft.Graph.Beta.Identity.Governance.inter nal.psm1:34 char:72 + ... th -Recurse -Include '*.ps1' -File | ForEach-Object { . $_.FullName } + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (:) [ForEach-Object], OutOfMemoryException + FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.ForEachObjectCommand

I know that I could use Azure Automation Hybrid Worker to run on a dedicated virtual machine. Obviously, this is not the idea of serverless infrastructure which is why I prefer to avoid this. I'm not sure if the memory limits of the Azure Automation sandbox will increase in the future or when (it likely will, after migration to Azure Container Instances was completed). However, this is not what I think would be a satisfactory solution as even today, memory consumption counts as it adds up on scale and Microsoft might expect lots of users running Microsoft Graph PowerShell on Azure Automation more and more.

I was wondering if there was anything wrong with the module as it is becoming simply bigger and bigger, and compared to other Graph modules, it is extremely large already. If it were by design today, is there anything you could do to improve size and memory consumption as such?

Screenshot comparing sizes of typical Microsoft Graph PowerShell module sizes in Azure Automation

Thanks! 😄

gavinbarron commented 3 months ago

We also have a size issue for Microsoft.Graph.Files in that during startup the peak memory usage exceeds 1GB and causes an OOM error in Azure Automation.

We should investigate the feasibility of either providing a manifest to allow for selective importing of cmdlets or omitting the help XML files and only providing online help to see if these approaches reduce the memory impact.

@timayabi2020 @petrhollayms

timayabi2020 commented 3 months ago

Issue also mentioned in #2475