search

PowerShell / PSResourceGet

PSResourceGet is the package manager for PowerShell
https://www.powershellgallery.com/packages/Microsoft.PowerShell.PSResourceGet
MIT License
487 stars 93 forks source link

`Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls? #1045

Open o-l-a-v opened 1 year ago

o-l-a-v commented 1 year ago

Summary of the new feature / enhancement

Doing PowerShellGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $InstalledModules is slow, seems like it queries one at a time.

The NuGet API can do multiple, like for instance (i know latest beta went away from using NuGet):

# Faster, multiple at a time
## Two manually
Invoke-RestMethod -Method 'Get' -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=IsLatestVersion and IsPrerelease eq false and (Id eq 'Az.Accounts' or Id eq 'Microsoft.Graph.Authentication')&semVerLevel=1.0.0"
## Ten first from a string array of module names
Invoke-RestMethod -Method 'Get' -Uri (
    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
            (
                $InstalledModules[0 .. 10].ForEach{
                    "Id eq '{0}'" -f $_
                } -join ' or '
            ) + ')&semVerLevel=1.0.0'
)

Here are some PowerShell to show how slow it is currently. I copied out a string array of module names.

Click to view ```powershell # String array of modules $Modules = [string[]]( 'AIPService,AWSPowerShell.NetCore,Az,Az.Accounts,Az.ADDomainServices,Az.Advisor,Az.Aks,Az.AlertsManagement,Az.AnalysisServices,Az.ApiManagement,Az.App,Az.AppConfiguration,Az.ApplicationInsights,Az.ApplicationMonitor,Az.Attestation,Az.Automanage,Az.Automation,Az.BareMetal,Az.Batch,Az.Billing,Az.BillingBenefits,Az.Blueprint,Az.BootStrapper,Az.BotService,Az.Cdn,Az.ChangeAnalysis,Az.CloudService,Az.CognitiveServices,Az.Communication,Az.Compute,Az.Compute.ManagedService,Az.ConfidentialLedger,Az.Confluent,Az.ConnectedKubernetes,Az.ConnectedMachine,Az.ConnectedNetwork,Az.ContainerInstance,Az.ContainerRegistry,Az.CosmosDB,Az.CostManagement,Az.CustomLocation,Az.CustomProviders,Az.Dashboard,Az.DataBox,Az.DataBoxEdge,Az.Databricks,Az.Datadog,Az.DataFactory,Az.DataLakeAnalytics,Az.DataLakeStore,Az.DataMigration,Az.DataProtection,Az.DataShare,Az.DedicatedHsm,Az.DeploymentManager,Az.DesktopVirtualization,Az.DeviceProvisioningServices,Az.DeviceUpdate,Az.DevSpaces,Az.DevTestLabs,Az.DigitalTwins,Az.DiskPool,Az.Dns,Az.DnsResolver,Az.DynatraceObservability,Az.EdgeOrder,Az.Elastic,Az.ElasticSan,Az.EventGrid,Az.EventHub,Az.FluidRelay,Az.FrontDoor,Az.Functions,Az.GuestConfiguration,Az.HanaOnAzure,Az.HDInsight,Az.HealthBot,Az.HealthcareApis,Az.HPCCache,Az.ImageBuilder,Az.ImportExport,Az.Insights,Az.IotCentral,Az.IotHub,Az.KeyVault,Az.KubernetesConfiguration,Az.Kusto,Az.LabServices,Az.LoadTesting,Az.LogicApp,Az.Logz,Az.MachineLearning,Az.MachineLearningCompute,Az.MachineLearningServices,Az.Maintenance,Az.ManagedServiceIdentity,Az.ManagedServices,Az.ManagementPartner,Az.Maps,Az.MariaDb,Az.Marketplace,Az.MarketplaceOrdering,Az.Media,Az.Migrate,Az.MixedReality,Az.MobileNetwork,Az.Monitor,Az.MonitoringSolutions,Az.MySql,Az.NetAppFiles,Az.Network,Az.NetworkFunction,Az.Nginx,Az.NotificationHubs,Az.OperationalInsights,Az.Orbital,Az.Peering,Az.PolicyInsights,Az.Portal,Az.PostgreSql,Az.PowerBIEmbedded,Az.PrivateDns,Az.Profile,Az.ProviderHub,Az.Purview,Az.Quota,Az.RecoveryServices,Az.RedisCache,Az.RedisEnterpriseCache,Az.Relay,Az.Reservations,Az.ResourceGraph,Az.ResourceMover,Az.Resources,Az.Search,Az.Security,Az.SecurityInsights,Az.ServiceBus,Az.ServiceFabric,Az.ServiceLinker,Az.SignalR,Az.SpringCloud,Az.Sql,Az.SqlVirtualMachine,Az.Ssh,Az.StackEdge,Az.StackHCI,Az.StackHCI.NetworkHUD,Az.Storage,Az.StorageMover,Az.StorageSync,Az.StreamAnalytics,Az.Subscription,Az.Support,Az.Synapse,Az.Tags,Az.TimeSeriesInsights,Az.Tools.Installer,Az.Tools.Migration,Az.Tools.Predictor,Az.TrafficManager,Az.VMware,Az.VoiceServices,Az.Websites,Az.WindowsIotServices,AzSK,AzSK.AAD,AzSK.ADO,AzSK.AzureDevOps,Azure,Azure.AnalysisServices,Azure.Storage,AzureAD,AzureADPreview,AzureRM.profile,AzViz,ConfluencePS,DefenderMAPS,Evergreen,ExchangeOnlineManagement,GetBIOS,ImportExcel,Intune.USB.Creator,IntuneBackupAndRestore,Invokeall,JWTDetails,Mailozaurr,Microsoft.Graph,Microsoft.Graph.Applications,Microsoft.Graph.Authentication,Microsoft.Graph.Bookings,Microsoft.Graph.Calendar,Microsoft.Graph.ChangeNotifications,Microsoft.Graph.CloudCommunications,Microsoft.Graph.Compliance,Microsoft.Graph.CrossDeviceExperiences,Microsoft.Graph.DeviceManagement,Microsoft.Graph.DeviceManagement.Actions,Microsoft.Graph.DeviceManagement.Administration,Microsoft.Graph.DeviceManagement.Enrolment,Microsoft.Graph.DeviceManagement.Functions,Microsoft.Graph.Devices.CloudPrint,Microsoft.Graph.Devices.CorporateManagement,Microsoft.Graph.Devices.ServiceAnnouncement,Microsoft.Graph.DirectoryObjects,Microsoft.Graph.Education,Microsoft.Graph.Files,Microsoft.Graph.Financials,Microsoft.Graph.Groups,Microsoft.Graph.Identity.DirectoryManagement,Microsoft.Graph.Identity.Governance,Microsoft.Graph.Identity.SignIns,Microsoft.Graph.Intune,Microsoft.Graph.Mail,Microsoft.Graph.ManagedTenants,Microsoft.Graph.Notes,Microsoft.Graph.People,Microsoft.Graph.PersonalContacts,Microsoft.Graph.Planner,Microsoft.Graph.Reports,Microsoft.Graph.SchemaExtensions,Microsoft.Graph.Search,Microsoft.Graph.Security,Microsoft.Graph.Sites,Microsoft.Graph.Teams,Microsoft.Graph.Users,Microsoft.Graph.Users.Actions,Microsoft.Graph.Users.Functions,Microsoft.Graph.WindowsUpdates,Microsoft.Online.SharePoint.PowerShell,Microsoft.PowerShell.ConsoleGuiTools,Microsoft.PowerShell.SecretManagement,Microsoft.PowerShell.SecretStore,Microsoft.RDInfra.RDPowershell,Microsoft.RdInfra.RDPowershell.Migration,MicrosoftGraphSecurity,MicrosoftPowerBIMgmt,MicrosoftPowerBIMgmt.Admin,MicrosoftPowerBIMgmt.Capacities,MicrosoftPowerBIMgmt.Data,MicrosoftPowerBIMgmt.Profile,MicrosoftPowerBIMgmt.Reports,MicrosoftPowerBIMgmt.Workspaces,MicrosoftTeams,MSAL.PS,MSGraphFunctions,MSOnline,Nevergreen,newtonsoft.json,Office365DnsChecker,Optimized.Mga,Optimized.Mga.AzureAD,Optimized.Mga.Mail,Optimized.Mga.Report,Optimized.Mga.SharePoint,PackageManagement,PartnerCenter,PartnerCenter.NetCore,Pester,platyPS,PnP.PowerShell,PolicyFileEditor,PoshRSJob,PowerShellGet,PSGraph,PSIntuneAuth,PSPackageProject,PSPKI,PSReadLine,PSScriptAnalyzer,PSWindowsUpdate,RunAsUser,SetBIOS,SharePointPnPPowerShellOnline,SHiPS,SpeculationControl,Trackyon.Utils,VSTeam,WindowsAutoPilotIntune'.Split(',') ) # Microsoft.PowerShell.PSResourceGet ## Get ### Using Cmdlet as is Measure-Command -Expression { $Script:PSResourceGetResults = [array]( Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $Modules | Sort-Object -Property 'Name' -Unique ) } ### Using `ForEach-Object -Parallel` Measure-Command -Expression { $Script:PSResourceGetResults = [array]( $Modules | ForEach-Object -Parallel { Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $_ } -ThrottleLimit 50 | Sort-Object -Property 'Name' -Unique ) } ## Present results $PSResourceGetResults.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'Name' 'Author' = [string] $_.'Author' 'Version' = [System.Version] $_.'Version' 'NormalizedVersion' = [System.Version] $_.'AdditionalMetadata'.'NormalizedVersion' } } | Format-Table -AutoSize # Bulk PowerShellGallery NuGet API ## Get Measure-Command -Expression { $PageSize = [byte] 30 $Page = [byte] 1 $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]]( $( do { $FromIndex = [uint16](($Page-1) * $PageSize) $ToIndex = [uint16]($Page * $PageSize -lt $Modules.'Count' ? $Page * $PageSize : $Modules.'Count') Invoke-RestMethod -Method 'Get' -Uri ( 'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' + ( $Modules[$FromIndex .. $ToIndex].ForEach{ "Id eq '{0}'" -f $_ } -join ' or ' ) + ')&semVerLevel=1.0.0' ) $Page++ } until ($ToIndex -ge $InstalledModules.'Count') ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique ) } ## Present results $PowerShellGalleryNuGetAPIResults.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'title'.'#text' 'Author' = [string] $_.'author'.'name' 'Version' = [System.Version] $_.'properties'.'Version' 'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion' } } | Format-Table -AutoSize ```

Proposed technical implementation details (optional)

SydneyhSmith commented 1 year ago

Thanks @o-l-a-v we have improvements planned

o-l-a-v commented 11 months ago

Did some experimenting with runspace pools for Find-PSResource on Az, Microsoft.Graph, Microsoft.Graph.Beta and all their dependencies, 166 unique modules in total.

We're talking big savings if both parallelizing and batching API requests. And both will work with Windows PowerShell.

Code if others want to experiment ```powershell # Function function Find-PSResourceInParallel { <# .SYNOPSIS Speed up PSResourceGet\Find-PSResource by parallizing using PowerShell native runspace factory. .NOTES Author: Olav Rønnestad Birkeland | github.com/o-l-a-v Created: 231116 Modified: 231116 .EXAMPLE Find-PSResourceInParallel -Type 'Module' -Name (Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name 'Az').'Dependencies'.'Name' #> [CmdletBinding()] [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]])] Param( [Parameter(Mandatory)] [string[]] $Name, [Parameter()] [ValidateNotNullOrEmpty()] [string] $PSResourceGetPath = (Get-Module -Name 'Microsoft.PowerShell.PSResourceGet').'Path', [Parameter()] [ValidateNotNullOrEmpty()] [string] $Repository = 'PSGallery', [Parameter()] [byte] $ThrottleLimit = 10, [Parameter()] [ValidateNotNullOrEmpty()] [ValidateSet('Module','Script')] [string] $Type = 'Module' ) # Begin Begin { # Assets $ScriptBlock = [scriptblock]{ [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo])] Param( [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Name, [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$PSResourceGetPath, [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Repository, [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Type ) $ErrorActionPreference = 'Stop' $null = Import-Module -Name $PSResourceGetPath Microsoft.PowerShell.PSResourceGet\Find-PSResource -Repository $Repository -Type $Type -Name $Name } # Initilize runspace pool $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$ThrottleLimit) $RunspacePool.Open() } # Process Process { # Start jobs in the runspace pool $RunspacePoolJobs = [PSCustomObject[]]( $( foreach ($ModuleName in $Name) { $PowerShellObject = [powershell]::Create().AddScript($ScriptBlock).AddParameters( @{ 'PSResourceGetPath' = [string] $PSResourceGetPath 'Repository' = [string] $Repository 'Type' = [string] $Type 'Name' = [string] $ModuleName } ) $PowerShellObject.'RunspacePool' = $RunspacePool [PSCustomObject]@{ 'ModuleName' = $ModuleName 'Instance' = $PowerShellObject 'Result' = $PowerShellObject.BeginInvoke() } } ) ) # Wait for jobs to finish $PrettyPrint = [string]('0'*$RunspacePoolJobs.'Count'.ToString().'Length') while ($RunspacePoolJobs.Where{-not $_.'Result'.'IsCompleted'}.'Count' -gt 0) { Write-Verbose -Message ( '{0} / {1} jobs finished, {2} / {0} was successfull.' -f ( $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted'}.'Count'.ToString($PrettyPrint), $RunspacePoolJobs.'Count'.ToString(), $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted' -and -not $_.'Instance'.'HadErrors'}.'Count'.ToString($PrettyPrint) ) ) Start-Sleep -Milliseconds 250 } # Get success state of jobs Write-Verbose -Message ( $RunspacePoolJobs.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'ModuleName' 'IsCompleted' = [bool] $_.'Result'.'IsCompleted' 'HadErrors' = [bool] $_.'Instance'.'HadErrors' } } | Sort-Object -Property 'ModuleName' | Format-Table | Out-String ) # Collect results $Results = [Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]]( $RunspacePoolJobs.ForEach{ $_.'Instance'.EndInvoke($_.'Result') } ) } # End End { # Terminate runspace pool $RunspacePool.Close() $RunspacePool.Dispose() # Output results $Results } } # Testing if ($false) { # Import PSResourceGet Import-Module -Name 'Microsoft.PowerShell.PSResourceGet' -RequiredVersion '1.0.1' # Assets $ListOfModules = [string[]]('Az','Microsoft.Graph','Microsoft.Graph.Beta') $ListOfModules += [string[]]((Find-PSResource -Name $ListOfModules).'Dependencies'.'Name') $ListOfModules = [string[]]($ListOfModules | Sort-Object -Unique) # Test Find-PSResourceInParallel -Name $ListOfModules -ThrottleLimit 20 Find-PSResourceInParallel -Name 'Az.*' # Measure / Compare ## Original Measure-Command -Expression { Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules } ## Parallel with Runspace Factory Measure-Command -Expression { Find-PSResourceInParallel -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules -ThrottleLimit 25 } ## Batch API manually sequentially ### Measure Measure-Command -Expression { $PageSize = [byte] 30 $Page = [byte] 1 $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]]( $( do { $FromIndex = [uint16](($Page-1) * $PageSize) $ToIndex = [uint16]($Page * $PageSize -lt $ListOfModules.'Count' ? $Page * $PageSize : $ListOfModules.'Count') Invoke-RestMethod -Method 'Get' -Uri ( 'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' + ( $ListOfModules[$FromIndex .. $ToIndex].ForEach{ "Id eq '{0}'" -f $_ } -join ' or ' ) + ')&semVerLevel=1.0.0' ) $Page++ } until ($ToIndex -ge $ListOfModules.'Count') ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique ) } ### Present results $PowerShellGalleryNuGetAPIResults.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'title'.'#text' 'Author' = [string] $_.'author'.'name' 'Version' = [System.Version] $_.'properties'.'Version' 'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion' } } | Format-Table -AutoSize } ```