search

PowerShell / PSResourceGet

PSResourceGet is the package manager for PowerShell
https://www.powershellgallery.com/packages/Microsoft.PowerShell.PSResourceGet
MIT License
494 stars 94 forks source link

`Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls? #1045

Open o-l-a-v opened 1 year ago

o-l-a-v commented 1 year ago

Summary of the new feature / enhancement

Doing PowerShellGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $InstalledModules is slow, seems like it queries one at a time.

The NuGet API can do multiple, like for instance (i know latest beta went away from using NuGet):

# Faster, multiple at a time
## Two manually
Invoke-RestMethod -Method 'Get' -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=IsLatestVersion and IsPrerelease eq false and (Id eq 'Az.Accounts' or Id eq 'Microsoft.Graph.Authentication')&semVerLevel=1.0.0"
## Ten first from a string array of module names
Invoke-RestMethod -Method 'Get' -Uri (
    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
            (
                $InstalledModules[0 .. 10].ForEach{
                    "Id eq '{0}'" -f $_
                } -join ' or '
            ) + ')&semVerLevel=1.0.0'
)

Here are some PowerShell to show how slow it is currently. I copied out a string array of module names.

Click to view ```powershell # String array of modules $Modules = [string[]]( 'AIPService,AWSPowerShell.NetCore,Az,Az.Accounts,Az.ADDomainServices,Az.Advisor,Az.Aks,Az.AlertsManagement,Az.AnalysisServices,Az.ApiManagement,Az.App,Az.AppConfiguration,Az.ApplicationInsights,Az.ApplicationMonitor,Az.Attestation,Az.Automanage,Az.Automation,Az.BareMetal,Az.Batch,Az.Billing,Az.BillingBenefits,Az.Blueprint,Az.BootStrapper,Az.BotService,Az.Cdn,Az.ChangeAnalysis,Az.CloudService,Az.CognitiveServices,Az.Communication,Az.Compute,Az.Compute.ManagedService,Az.ConfidentialLedger,Az.Confluent,Az.ConnectedKubernetes,Az.ConnectedMachine,Az.ConnectedNetwork,Az.ContainerInstance,Az.ContainerRegistry,Az.CosmosDB,Az.CostManagement,Az.CustomLocation,Az.CustomProviders,Az.Dashboard,Az.DataBox,Az.DataBoxEdge,Az.Databricks,Az.Datadog,Az.DataFactory,Az.DataLakeAnalytics,Az.DataLakeStore,Az.DataMigration,Az.DataProtection,Az.DataShare,Az.DedicatedHsm,Az.DeploymentManager,Az.DesktopVirtualization,Az.DeviceProvisioningServices,Az.DeviceUpdate,Az.DevSpaces,Az.DevTestLabs,Az.DigitalTwins,Az.DiskPool,Az.Dns,Az.DnsResolver,Az.DynatraceObservability,Az.EdgeOrder,Az.Elastic,Az.ElasticSan,Az.EventGrid,Az.EventHub,Az.FluidRelay,Az.FrontDoor,Az.Functions,Az.GuestConfiguration,Az.HanaOnAzure,Az.HDInsight,Az.HealthBot,Az.HealthcareApis,Az.HPCCache,Az.ImageBuilder,Az.ImportExport,Az.Insights,Az.IotCentral,Az.IotHub,Az.KeyVault,Az.KubernetesConfiguration,Az.Kusto,Az.LabServices,Az.LoadTesting,Az.LogicApp,Az.Logz,Az.MachineLearning,Az.MachineLearningCompute,Az.MachineLearningServices,Az.Maintenance,Az.ManagedServiceIdentity,Az.ManagedServices,Az.ManagementPartner,Az.Maps,Az.MariaDb,Az.Marketplace,Az.MarketplaceOrdering,Az.Media,Az.Migrate,Az.MixedReality,Az.MobileNetwork,Az.Monitor,Az.MonitoringSolutions,Az.MySql,Az.NetAppFiles,Az.Network,Az.NetworkFunction,Az.Nginx,Az.NotificationHubs,Az.OperationalInsights,Az.Orbital,Az.Peering,Az.PolicyInsights,Az.Portal,Az.PostgreSql,Az.PowerBIEmbedded,Az.PrivateDns,Az.Profile,Az.ProviderHub,Az.Purview,Az.Quota,Az.RecoveryServices,Az.RedisCache,Az.RedisEnterpriseCache,Az.Relay,Az.Reservations,Az.ResourceGraph,Az.ResourceMover,Az.Resources,Az.Search,Az.Security,Az.SecurityInsights,Az.ServiceBus,Az.ServiceFabric,Az.ServiceLinker,Az.SignalR,Az.SpringCloud,Az.Sql,Az.SqlVirtualMachine,Az.Ssh,Az.StackEdge,Az.StackHCI,Az.StackHCI.NetworkHUD,Az.Storage,Az.StorageMover,Az.StorageSync,Az.StreamAnalytics,Az.Subscription,Az.Support,Az.Synapse,Az.Tags,Az.TimeSeriesInsights,Az.Tools.Installer,Az.Tools.Migration,Az.Tools.Predictor,Az.TrafficManager,Az.VMware,Az.VoiceServices,Az.Websites,Az.WindowsIotServices,AzSK,AzSK.AAD,AzSK.ADO,AzSK.AzureDevOps,Azure,Azure.AnalysisServices,Azure.Storage,AzureAD,AzureADPreview,AzureRM.profile,AzViz,ConfluencePS,DefenderMAPS,Evergreen,ExchangeOnlineManagement,GetBIOS,ImportExcel,Intune.USB.Creator,IntuneBackupAndRestore,Invokeall,JWTDetails,Mailozaurr,Microsoft.Graph,Microsoft.Graph.Applications,Microsoft.Graph.Authentication,Microsoft.Graph.Bookings,Microsoft.Graph.Calendar,Microsoft.Graph.ChangeNotifications,Microsoft.Graph.CloudCommunications,Microsoft.Graph.Compliance,Microsoft.Graph.CrossDeviceExperiences,Microsoft.Graph.DeviceManagement,Microsoft.Graph.DeviceManagement.Actions,Microsoft.Graph.DeviceManagement.Administration,Microsoft.Graph.DeviceManagement.Enrolment,Microsoft.Graph.DeviceManagement.Functions,Microsoft.Graph.Devices.CloudPrint,Microsoft.Graph.Devices.CorporateManagement,Microsoft.Graph.Devices.ServiceAnnouncement,Microsoft.Graph.DirectoryObjects,Microsoft.Graph.Education,Microsoft.Graph.Files,Microsoft.Graph.Financials,Microsoft.Graph.Groups,Microsoft.Graph.Identity.DirectoryManagement,Microsoft.Graph.Identity.Governance,Microsoft.Graph.Identity.SignIns,Microsoft.Graph.Intune,Microsoft.Graph.Mail,Microsoft.Graph.ManagedTenants,Microsoft.Graph.Notes,Microsoft.Graph.People,Microsoft.Graph.PersonalContacts,Microsoft.Graph.Planner,Microsoft.Graph.Reports,Microsoft.Graph.SchemaExtensions,Microsoft.Graph.Search,Microsoft.Graph.Security,Microsoft.Graph.Sites,Microsoft.Graph.Teams,Microsoft.Graph.Users,Microsoft.Graph.Users.Actions,Microsoft.Graph.Users.Functions,Microsoft.Graph.WindowsUpdates,Microsoft.Online.SharePoint.PowerShell,Microsoft.PowerShell.ConsoleGuiTools,Microsoft.PowerShell.SecretManagement,Microsoft.PowerShell.SecretStore,Microsoft.RDInfra.RDPowershell,Microsoft.RdInfra.RDPowershell.Migration,MicrosoftGraphSecurity,MicrosoftPowerBIMgmt,MicrosoftPowerBIMgmt.Admin,MicrosoftPowerBIMgmt.Capacities,MicrosoftPowerBIMgmt.Data,MicrosoftPowerBIMgmt.Profile,MicrosoftPowerBIMgmt.Reports,MicrosoftPowerBIMgmt.Workspaces,MicrosoftTeams,MSAL.PS,MSGraphFunctions,MSOnline,Nevergreen,newtonsoft.json,Office365DnsChecker,Optimized.Mga,Optimized.Mga.AzureAD,Optimized.Mga.Mail,Optimized.Mga.Report,Optimized.Mga.SharePoint,PackageManagement,PartnerCenter,PartnerCenter.NetCore,Pester,platyPS,PnP.PowerShell,PolicyFileEditor,PoshRSJob,PowerShellGet,PSGraph,PSIntuneAuth,PSPackageProject,PSPKI,PSReadLine,PSScriptAnalyzer,PSWindowsUpdate,RunAsUser,SetBIOS,SharePointPnPPowerShellOnline,SHiPS,SpeculationControl,Trackyon.Utils,VSTeam,WindowsAutoPilotIntune'.Split(',') ) # Microsoft.PowerShell.PSResourceGet ## Get ### Using Cmdlet as is Measure-Command -Expression { $Script:PSResourceGetResults = [array]( Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $Modules | Sort-Object -Property 'Name' -Unique ) } ### Using `ForEach-Object -Parallel` Measure-Command -Expression { $Script:PSResourceGetResults = [array]( $Modules | ForEach-Object -Parallel { Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $_ } -ThrottleLimit 50 | Sort-Object -Property 'Name' -Unique ) } ## Present results $PSResourceGetResults.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'Name' 'Author' = [string] $_.'Author' 'Version' = [System.Version] $_.'Version' 'NormalizedVersion' = [System.Version] $_.'AdditionalMetadata'.'NormalizedVersion' } } | Format-Table -AutoSize # Bulk PowerShellGallery NuGet API ## Get Measure-Command -Expression { $PageSize = [byte] 30 $Page = [byte] 1 $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]]( $( do { $FromIndex = [uint16](($Page-1) * $PageSize) $ToIndex = [uint16]($Page * $PageSize -lt $Modules.'Count' ? $Page * $PageSize : $Modules.'Count') Invoke-RestMethod -Method 'Get' -Uri ( 'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' + ( $Modules[$FromIndex .. $ToIndex].ForEach{ "Id eq '{0}'" -f $_ } -join ' or ' ) + ')&semVerLevel=1.0.0' ) $Page++ } until ($ToIndex -ge $InstalledModules.'Count') ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique ) } ## Present results $PowerShellGalleryNuGetAPIResults.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'title'.'#text' 'Author' = [string] $_.'author'.'name' 'Version' = [System.Version] $_.'properties'.'Version' 'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion' } } | Format-Table -AutoSize ```

Proposed technical implementation details (optional)

SydneyhSmith commented 1 year ago

Thanks @o-l-a-v we have improvements planned

o-l-a-v commented 1 year ago

Did some experimenting with runspace pools for Find-PSResource on Az, Microsoft.Graph, Microsoft.Graph.Beta and all their dependencies, 166 unique modules in total.

We're talking big savings if both parallelizing and batching API requests. And both will work with Windows PowerShell.

Code if others want to experiment ```powershell # Function function Find-PSResourceInParallel { <# .SYNOPSIS Speed up PSResourceGet\Find-PSResource by parallizing using PowerShell native runspace factory. .NOTES Author: Olav Rønnestad Birkeland | github.com/o-l-a-v Created: 231116 Modified: 231116 .EXAMPLE Find-PSResourceInParallel -Type 'Module' -Name (Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name 'Az').'Dependencies'.'Name' #> [CmdletBinding()] [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]])] Param( [Parameter(Mandatory)] [string[]] $Name, [Parameter()] [ValidateNotNullOrEmpty()] [string] $PSResourceGetPath = (Get-Module -Name 'Microsoft.PowerShell.PSResourceGet').'Path', [Parameter()] [ValidateNotNullOrEmpty()] [string] $Repository = 'PSGallery', [Parameter()] [byte] $ThrottleLimit = 10, [Parameter()] [ValidateNotNullOrEmpty()] [ValidateSet('Module','Script')] [string] $Type = 'Module' ) # Begin Begin { # Assets $ScriptBlock = [scriptblock]{ [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo])] Param( [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Name, [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$PSResourceGetPath, [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Repository, [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Type ) $ErrorActionPreference = 'Stop' $null = Import-Module -Name $PSResourceGetPath Microsoft.PowerShell.PSResourceGet\Find-PSResource -Repository $Repository -Type $Type -Name $Name } # Initilize runspace pool $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$ThrottleLimit) $RunspacePool.Open() } # Process Process { # Start jobs in the runspace pool $RunspacePoolJobs = [PSCustomObject[]]( $( foreach ($ModuleName in $Name) { $PowerShellObject = [powershell]::Create().AddScript($ScriptBlock).AddParameters( @{ 'PSResourceGetPath' = [string] $PSResourceGetPath 'Repository' = [string] $Repository 'Type' = [string] $Type 'Name' = [string] $ModuleName } ) $PowerShellObject.'RunspacePool' = $RunspacePool [PSCustomObject]@{ 'ModuleName' = $ModuleName 'Instance' = $PowerShellObject 'Result' = $PowerShellObject.BeginInvoke() } } ) ) # Wait for jobs to finish $PrettyPrint = [string]('0'*$RunspacePoolJobs.'Count'.ToString().'Length') while ($RunspacePoolJobs.Where{-not $_.'Result'.'IsCompleted'}.'Count' -gt 0) { Write-Verbose -Message ( '{0} / {1} jobs finished, {2} / {0} was successfull.' -f ( $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted'}.'Count'.ToString($PrettyPrint), $RunspacePoolJobs.'Count'.ToString(), $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted' -and -not $_.'Instance'.'HadErrors'}.'Count'.ToString($PrettyPrint) ) ) Start-Sleep -Milliseconds 250 } # Get success state of jobs Write-Verbose -Message ( $RunspacePoolJobs.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'ModuleName' 'IsCompleted' = [bool] $_.'Result'.'IsCompleted' 'HadErrors' = [bool] $_.'Instance'.'HadErrors' } } | Sort-Object -Property 'ModuleName' | Format-Table | Out-String ) # Collect results $Results = [Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]]( $RunspacePoolJobs.ForEach{ $_.'Instance'.EndInvoke($_.'Result') } ) } # End End { # Terminate runspace pool $RunspacePool.Close() $RunspacePool.Dispose() # Output results $Results } } # Testing if ($false) { # Import PSResourceGet Import-Module -Name 'Microsoft.PowerShell.PSResourceGet' -RequiredVersion '1.0.1' # Assets $ListOfModules = [string[]]('Az','Microsoft.Graph','Microsoft.Graph.Beta') $ListOfModules += [string[]]((Find-PSResource -Name $ListOfModules).'Dependencies'.'Name') $ListOfModules = [string[]]($ListOfModules | Sort-Object -Unique) # Test Find-PSResourceInParallel -Name $ListOfModules -ThrottleLimit 20 Find-PSResourceInParallel -Name 'Az.*' # Measure / Compare ## Original Measure-Command -Expression { Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules } ## Parallel with Runspace Factory Measure-Command -Expression { Find-PSResourceInParallel -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules -ThrottleLimit 25 } ## Batch API manually sequentially ### Measure Measure-Command -Expression { $PageSize = [byte] 30 $Page = [byte] 1 $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]]( $( do { $FromIndex = [uint16](($Page-1) * $PageSize) $ToIndex = [uint16]($Page * $PageSize -lt $ListOfModules.'Count' ? $Page * $PageSize : $ListOfModules.'Count') Invoke-RestMethod -Method 'Get' -Uri ( 'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' + ( $ListOfModules[$FromIndex .. $ToIndex].ForEach{ "Id eq '{0}'" -f $_ } -join ' or ' ) + ')&semVerLevel=1.0.0' ) $Page++ } until ($ToIndex -ge $ListOfModules.'Count') ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique ) } ### Present results $PowerShellGalleryNuGetAPIResults.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'title'.'#text' 'Author' = [string] $_.'author'.'name' 'Version' = [System.Version] $_.'properties'.'Version' 'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion' } } | Format-Table -AutoSize } ```
o-l-a-v commented 3 weeks ago

If one only cares about getting the latest stable version of a lot of modules from the PowerShell Gallery, I've found that one can safely ask the API about 30 modules in one request.

Here's a function I use in my PowerShell modules updater script ( https://github.com/o-l-a-v/PowerShell-Projects/tree/master/PowerShellModulesUpdater ) which takes few seconds to find latest version of 358 modules.

Testing:

# Get latest version of all installed modules
## Assets
$InstalledPSResources = [string[]](
    (Get-InstalledPSResource -Path ('{0}\Microsoft\PowerShell\Modules' -f $env:LOCALAPPDATA)).'Name'
)

## One by one
$Stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Find-PSResource -Name $InstalledPSResources -Repository 'PSGallery'
$Stopwatch.Stop(); $Stopwatch.'Elapsed'.ToString()

## This function
$Stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds $InstalledPSResources -MinimalInfo
$Stopwatch.Stop(); $Stopwatch.'Elapsed'.ToString()
Click to view ```pwsh function Find-PSGalleryPackageLatestVersionUsingApiInBatch { <# .SYNOPSIS Get PowerShell Gallery package version info for one or multiple packages using the API directly. .NOTES Author: Olav Rønnestad Birkeland | github.com/o-l-a-v Created: 2024-03-13 Modified: 2024-04-01 .EXAMPLE . $psEditor.GetEditorContext().CurrentFile.Path Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds 'Az.Accounts','Az.Resources' -Verbose Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds 'Az.CosmosDB' -IncludePrerelease -Verbose Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds 'Az.*' -MinimalInfo -Verbose Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds 'Az.*' -AsHashtable -Verbose Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds 'Az.*' -AsHashtable -MinimalInfo -Verbose Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds 'Az.*' -MinimalInfo -Verbose .EXAMPLE # Get latest version of all installed modules ## Assets $InstalledPSResources = [string[]]( (Get-InstalledPSResource -Path ('{0}\Microsoft\PowerShell\Modules' -f $env:LOCALAPPDATA)).'Name' ) ## One by one $Stopwatch = [System.Diagnostics.Stopwatch]::StartNew() Find-PSResource -Name $InstalledPSResources -Repository 'PSGallery' $Stopwatch.Stop(); $Stopwatch.'Elapsed'.ToString() ## This function $Stopwatch = [System.Diagnostics.Stopwatch]::StartNew() Find-PSGalleryPackageLatestVersionUsingApiInBatch -PackageIds $InstalledPSResources -MinimalInfo $Stopwatch.Stop(); $Stopwatch.'Elapsed'.ToString() #> # Input parameters and expected output [CmdletBinding(DefaultParameterSetName='Stable')] [CmdletBinding()] [OutputType([hashtable],[PSCustomObject])] Param( [Parameter(Mandatory, ParameterSetName = 'Absolute')] [Parameter(Mandatory, ParameterSetName = 'Stable')] [Parameter(Mandatory)] $PackageIds, [Alias('Absolute','Prerelease')] [Parameter(HelpMessage = 'Include prerelease.', ParameterSetName = 'Absolute')] [switch] $IncludePrerelease, [Parameter(HelpMessage = 'Optionally return result as hashtable.', ParameterSetName = 'Absolute')] [Parameter(HelpMessage = 'Optionally return result as hashtable.', ParameterSetName = 'Stable')] [switch] $AsHashtable, [Parameter(HelpMessage = 'Strip some info to speed up interaction with PowerShell Gallery API.', ParameterSetName = 'Absolute')] [Parameter(HelpMessage = 'Strip some info to speed up interaction with PowerShell Gallery API.', ParameterSetName = 'Stable')] [switch] $MinimalInfo ) # Process Process { # Parse and validate input ## Is of expected type - String or string array if ($PackageIds -is [string] -or $PackageIds.ForEach{$_ -is [string]} -notcontains $false) { Write-Verbose -Message 'Input seems legit.' } else { Throw [ArgumentException]::new('Input is not a string or string array.') } ## Cast input to string array, remove duplicates and sort alphabetically $PackageIdsFiltered = [string[]]($PackageIds -as [string[]] | Where-Object -FilterScript {-not [string]::IsNullOrEmpty($_)} | Sort-Object -Unique) ## Verify we still have any input if ($PackageIdsFiltered.Where{-not[string]::IsNullOrEmpty($_)}.'Count' -le 0) { Throw [ArgumentException]::new('Input seems to be empty.') } ## Verify that input is as expected if ($PackageIdsFiltered.Where({$_.'Length' -gt 90 -or $_ -notmatch '^\*$|^(\*?)([a-zA-Z0-9.\-_]{1,})(\*?)$'},'First').'Count' -gt 0) { Throw [ArgumentException]::new('Input contains unexpected characters or is too long (>90ch).') } ## Output warning if duplicates was found if ($PackageIdsFiltered.'Count' -lt $PackageIds.'Count') { Write-Warning -Message 'Input $PackageIds had duplicates / multiple items with the same value.' } # Create API filter based on parameter set name $VersionFilter = [string]( $( if ($IncludePrerelease.'IsPresent') { 'IsAbsoluteLatestVersion' } else { 'IsLatestVersion and not IsPrerelease' } ) ) if ($MinimalInfo.'IsPresent') { $Select = [string] '&$select=Authors,Published,RequireLicenseAcceptance,Tags,Version' -f $Filter } # Skip batching if input is only one module if ($PackageIdsFiltered.'Count' -eq 1 -and $PackageIdsFiltered.Where{$_.Contains('*')}.'Count' -le 0) { Write-Verbose -Message 'Skipping batching because input is one specific package ID.' $Results = [System.Xml.XmlElement[]]( $PackageIdsFiltered.ForEach{ $Uri = [string]( "https://www.powershellgallery.com/api/v2/FindPackagesById()?id='{0}'&`$filter={1}{2}&semVerLevel=1.0.0" -f ( $_, $VersionFilter, $Select ) ) Write-Verbose -Message $Uri Invoke-RestMethod -Method 'Get' -Uri $Uri } ) } # Else - Do batching, paging etc. else { # Assets ## PowerShell version - To enable version specific features $IsPowerShell72 = [bool]($PSVersionTable.'PSVersion' -ge $([System.Version]('7.2'))) $IsPowerShell74 = [bool]($PSVersionTable.'PSVersion' -ge $([System.Version]('7.4'))) ## PowerShell Gallery API $ApiPageSize = [byte] 100 $Headers = [ordered]@{ 'Accept' = [string] 'application/atom+xml;charset=UTF-8' 'Accept-Encoding' = [string] 'gzip, deflate' } if ($IsPowerShell74) { $Headers.'Accept-Encoding' = [string] '{0}, br' -f $Headers.'Accept-Encoding' } ## Batching of input $ArrayLastIndex = [uint16]($PackageIdsFiltered.'Count'-1) $ArrayPage = [byte] 0 $ArrayPageSize = [byte] 30 ## Results $Results = [System.Collections.Generic.List[System.Xml.XmlElement]]::new() # Find packages Write-Verbose -Message ('Headers:{0}' -f (ConvertTo-Json -Depth 1 -InputObject $Headers -Compress)) do { $ArrayCurrentLastIndex = [uint16]($([uint16[]]($ArrayLastIndex,(($ArrayPage*$ArrayPageSize)+$ArrayPageSize-1) | Sort-Object))[0]) $ApiPage = [byte] 0 do { # Create variable to splat into Invoke-RestMethod $Splat = [ordered]@{ 'Headers' = $Headers 'Method' = [string] 'Get' 'Uri' = [string]( ('https://www.powershellgallery.com/api/v2/Packages?$filter={0} and (' -f $VersionFilter) + ( $PackageIdsFiltered[($ArrayPage*$ArrayPageSize) .. $ArrayCurrentLastIndex].ForEach{ if ($_.EndsWith('*') -and $_.StartsWith('*')) { "substringof('{0}',Id)" -f $_.Replace('*','') } elseif ($_.EndsWith('*')) { "startswith(Id,'{0}')" -f $_.Replace('*','') } elseif ($_.StartsWith('*')) { "endswith(Id,'{0}')" -f $_.Replace('*','') } else { "Id eq '{0}'" -f $_ } } -join ' or ' ) + ('){0}&semVerLevel=1.0.0$inlinecount=allpages&$skip={1}&$top={2}' -f $Select, ($ApiPage*$ApiPageSize), $ApiPageSize) ) } # Use retry and WebSession if PowerShell >= 7.2 if ($IsPowerShell72) { if ($ApiPage -le 0 -and $ArrayPage -le 0) { $Splat.'SessionVariable' = [string] 'WebSession' } else { $Splat.'WebSession' = $WebSession } $Splat.'HttpVersion' = [System.Version] '2.0' $Splat.'MaximumRetryCount' = [int32] 2 $Splat.'RetryIntervalSec' = [int32] 1 } # Do the request Write-Verbose -Message ('{0}:{1}' -f ($ArrayPage+1).ToString('00'), $Splat.'Uri') $Response = [System.Xml.XmlElement[]](Invoke-RestMethod @Splat) if ($Response.'Count' -gt 0) { $Results.AddRange($Response) } $ApiPage++ } while ($Response.'Count' -gt 0 -and $Response.'Count' % $ApiPageSize -eq 0) $ArrayPage++ } while ($ArrayCurrentLastIndex -lt $ArrayLastIndex) } # Parse and create output Write-Verbose -Message 'Parse and create output' $OutputAsPSCustomObjectArray = [PSCustomObject[]]( $( $Results.Where{ -not [string]::IsNullOrEmpty($_.'Id') }.ForEach{ $Tags = [string[]]( $( if ($_.'properties'.'Tags' -is [System.Xml.XmlLinkedNode]) { $_.'properties'.'Tags'.'#text' } else { $_.'properties'.'Tags' } ).Split(' ', [StringSplitOptions]::RemoveEmptyEntries) | Sort-Object -Unique ) [PSCustomObject]@{ 'Name' = [string] $_.'title'.'#text' 'Author' = [string] $_.'author'.'name' 'Depencencies' = [string[]]( $( if (-not [string]::IsNullOrEmpty($_.'properties'.'Dependencies')) { $_.'properties'.'Dependencies'.Split( '|', [StringSplitOptions]::RemoveEmptyEntries ).ForEach{ $_.Split(':', [StringSplitOptions]::RemoveEmptyEntries)[0] } | Sort-Object -Unique } ) ) 'Owners' = [string] $_.'properties'.'owners' 'PublishedDate' = [nullable[datetime]] $_.'properties'.'Published'.'#text' 'RequireLicenseAcceptance' = [bool]($_.'properties'.'RequireLicenseAcceptance'.'#text' -eq 'true') 'Tags' = [string[]] $Tags 'Type' = [string]$( if ($Tags.Contains('PSModule')){'Module'} elseif ($Tags.Contains('PSScript')){'Script'} else {'Unknown'} ) 'Unlisted' = [bool]$( [string]::IsNullOrEmpty($_.'properties'.'Published'.'#text') -or $([datetime]($_.'properties'.'Published'.'#text')).'Year' -le 1900 ) 'Version' = [System.Version] $_.'properties'.'Version'.Split('-')[0] 'Prerelease' = [string]$(if($_.'properties'.'Version'.Contains('-')){$_.'properties'.'Version'.Split('-')[-1]}) 'VersionAsString' = [string] $_.'properties'.'Version' } } ) | Sort-Object -Property 'Name' ) # Return if ($AsHashtable) { Write-Verbose -Message 'Return output as hashtable' $OutputAsHashtable = [hashtable]@{} $OutputAsPSCustomObjectArray.ForEach{ $OutputAsHashtable.Add($_.'Name',$_) } $OutputAsHashtable } else { Write-Verbose -Message 'Return output as PSCustomObject' $OutputAsPSCustomObjectArray.ForEach{ $_ } } } } ```

Benefits with this approach includes less strain on the API. If this could be implemented in C# for PSResourceGet we speak MASSIVE gains in performance.