Speed up NuGet v3 queries by caching and/or ETag

o-l-a-v commented 8 months ago

Edit: I assumed wrong in my OP, but there are other things that can be done to speed up NuGet v3 queries described in following comment:

https://github.com/PowerShell/PSResourceGet/issues/1603#issuecomment-2028869625

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.

Steps to reproduce

I'm experimenting with pwsh.gallery + PSResourceGet.

Added it as repository like so:

Register-PSResourceRepository -Name 'pwsh.gallery' -Uri 'https://pwsh.gallery/index.json'

Then I wanted to see how fast it can return latest version of a given module. I did:

Find-PSResource -Repository 'pwsh.gallery' -Name 'Az.Accounts' -Verbose -Debug

I notice that PSResourceGet does three API calls where the first and last seems totally unneccessary for getting latest version.

One can determine latest version by just getting https://pwsh.gallery/az.accounts/index.json, or one could go straight to https://pwsh.gallery/az.accounts/page/latest.json.

PS > Find-PSResource -Repository 'pwsh.gallery' -Name 'Az.Accounts' -Verbose -Debug
DEBUG: In FindPSResource::ProcessResourceNameParameterSet()
DEBUG: Filtering package name(s) on wildcards
DEBUG: In FindHelper::FindByResourceName()
DEBUG: Parameters passed in >>> Name: 'Az.Accounts'; ResourceType: 'None'; VersionRange: ''; NuGetVersion: ''; VersionType: 'NoVersion'; Version: ''; Prerelease: 'False'; Tag: ''; Repository: 'pwsh.gallery'; IncludeDependencies 'False'
DEBUG: Searching through repository 'pwsh.gallery'
DEBUG: In FindHelper::SearchByNames()
DEBUG: No version specified, package name is specified
DEBUG: In V3ServerAPICalls::FindName()
DEBUG: In V3ServerAPICalls::FindNameHelper()
DEBUG: In V3ServerAPICalls::GetVersionedPackageEntriesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::GetResourcesFromServiceIndex()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/index.json'
DEBUG: In V3ServerAPICalls::FindRegistrationsBaseUrl()
DEBUG: In V3ServerAPICalls::GetVersionedResponsesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/index.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementsFromResponse()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/page/recent.json'
DEBUG: In V3ServerAPICalls::IsLatestVersionFirstForSearch()
DEBUG: 'Az.Accounts' version parsed as '2.16.0'
DEBUG: Found package 'Az.Accounts' version '2.16.0'
DEBUG: Package 'Az.Accounts' returned from server
DEBUG: Package 'Az.Accounts' was previously discovered and returned

Name        Version Prerelease Repository   Description
----        ------- ---------- ----------   -----------
Az.Accounts 2.16.0             pwsh.gallery

PS >

Questions:

Wouldn't it be better to just get <nugetv3api>/az.accounts/index.json or <nugetv3api>/az.accounts/page/latest.json?
Would it make sense to support -Version 'Latest' as input to Find-PSResource?

Expected behavior

When getting latest version only, Find-PSResource does just enough API calls, to 1) speed up end user experience, and 2) not put unneccessary load on the target repository/API.

Actual behavior

Find-PSResource gets <nugetv3api>/az.accounts/page/recent.json when getting latest version only.

Error details

No response

Environment data

PowerShell v7.4.1 x64 on Windows 11 23H2
Microsoft.PowerShell.PSResourceGet v1.0.3

Visuals

No response

o-l-a-v commented 8 months ago

If asking for two packages PSResourceGet seems to do a total of six API calls.

PS > Find-PSResource -Repository 'pwsh.gallery' -Name 'Az.Accounts','Az.Resources' -Verbose -Debug
DEBUG: In FindPSResource::ProcessResourceNameParameterSet()
DEBUG: Filtering package name(s) on wildcards
DEBUG: In FindHelper::FindByResourceName()
DEBUG: Parameters passed in >>> Name: 'Az.Accounts,Az.Resources'; ResourceType: 'None'; VersionRange: ''; NuGetVersion: ''; VersionType: 'NoVersion'; Version: ''; Prerelease: 'False'; Tag: ''; Repository: 'pwsh.gallery'; IncludeDependencies 'False'
DEBUG: Searching through repository 'pwsh.gallery'
DEBUG: In FindHelper::SearchByNames()
DEBUG: No version specified, package name is specified
DEBUG: In V3ServerAPICalls::FindName()
DEBUG: In V3ServerAPICalls::FindNameHelper()
DEBUG: In V3ServerAPICalls::GetVersionedPackageEntriesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::GetResourcesFromServiceIndex()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/index.json'
DEBUG: In V3ServerAPICalls::FindRegistrationsBaseUrl()
DEBUG: In V3ServerAPICalls::GetVersionedResponsesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/index.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementsFromResponse()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/page/recent.json'
DEBUG: In V3ServerAPICalls::IsLatestVersionFirstForSearch()
DEBUG: 'Az.Accounts' version parsed as '2.16.0'
DEBUG: Found package 'Az.Accounts' version '2.16.0'
DEBUG: Package 'Az.Accounts' returned from server
DEBUG: Package 'Az.Accounts' was previously discovered and returned

DEBUG: No version specified, package name is specified
DEBUG: In V3ServerAPICalls::FindName()
DEBUG: In V3ServerAPICalls::FindNameHelper()
DEBUG: In V3ServerAPICalls::GetVersionedPackageEntriesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::GetResourcesFromServiceIndex()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/index.json'
DEBUG: In V3ServerAPICalls::FindRegistrationsBaseUrl()
DEBUG: In V3ServerAPICalls::GetVersionedResponsesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.resources/index.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementsFromResponse()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.resources/page/recent.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.resources/page/older.json'
DEBUG: In V3ServerAPICalls::IsLatestVersionFirstForSearch()
DEBUG: 'Az.Resources' version parsed as '7.0.0-preview'
DEBUG: 'Az.Resources' version parsed as '6.16.0'
DEBUG: Found package 'Az.Resources' version '6.16.0'
DEBUG: Package 'Az.Resources' returned from server
DEBUG: Package 'Az.Resources' was previously discovered and returned
Name         Version Prerelease Repository   Description
----         ------- ---------- ----------   -----------
Az.Accounts  2.16.0             pwsh.gallery
Az.Resources 6.16.0             pwsh.gallery

PS >

PSResourceGet seems to be better optimized with nuget v2 / regular PowerShell Gallery:

PS > Find-PSResource -Repository 'PSGallery' -Name 'Az.Accounts','Az.Resources' -Verbose -Debug
DEBUG: In FindPSResource::ProcessResourceNameParameterSet()
DEBUG: Filtering package name(s) on wildcards
DEBUG: In FindHelper::FindByResourceName()
DEBUG: Parameters passed in >>> Name: 'Az.Accounts,Az.Resources'; ResourceType: 'None'; VersionRange: ''; NuGetVersion: ''; VersionType: 'NoVersion'; Version: ''; Prerelease: 'False'; Tag: ''; Repository: 'PSGallery'; IncludeDependencies 'False'
DEBUG: Searching through repository 'PSGallery'
DEBUG: In FindHelper::SearchByNames()
DEBUG: No version specified, package name is specified
DEBUG: In V2ServerAPICalls::FindName()
DEBUG: In V2ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'&$inlinecount=allpages&$filter=IsLatestVersion and Id eq 'Az.Accounts''
DEBUG: Found package 'Az.Accounts' version '2.16.0'
DEBUG: Package 'Az.Accounts' returned from server
DEBUG: Package 'Az.Accounts' was previously discovered and returned

DEBUG: No version specified, package name is specified
DEBUG: In V2ServerAPICalls::FindName()
DEBUG: In V2ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Resources'&$inlinecount=allpages&$filter=IsLatestVersion and Id eq 'Az.Resources''
DEBUG: Found package 'Az.Resources' version '6.16.0'
DEBUG: Package 'Az.Resources' returned from server
DEBUG: Package 'Az.Resources' was previously discovered and returned
Name         Version Prerelease Repository Description
----         ------- ---------- ---------- -----------
Az.Accounts  2.16.0             PSGallery  Microsoft Azure PowerShell - Accounts credential management cmdlets for Azure Resource Manager in Windows PowerShell and PowerShell Core.…
Az.Resources 6.16.0             PSGallery  Microsoft Azure PowerShell - Azure Resource Manager and Active Directory cmdlets in Windows PowerShell and PowerShell Core.  Manages subscriptions, tenants, resource groups, deployment templates, providers,…

PS >

o-l-a-v commented 7 months ago

Other nuget v3 API's than pwsh.gallery seems to act different. Take nuget.org for instance. It does not have <nugetv3api>/<package>/page/latest.json.

Example: NuGet.Versioning:

https://www.nuget.org/packages/NuGet.Versioning
index.json (works): https://api.nuget.org/v3/registration5-gz-semver2/nuget.versioning/index.json
Latest version as of writing (works): https://api.nuget.org/v3/registration5-gz-semver2/nuget.versioning/page/6.1.0/6.9.1.json
page/recent.json (does not work): https://api.nuget.org/v3/registration5-gz-semver2/nuget.versioning/page/recent.json

So maybe this isn't feasible with nuget v3 APIs after all.

JustinGrote commented 7 months ago

@o-l-a-v the v3 protocol specifically states you have to first look up the registration (which you can cache), then use the registration to find the package index, then use the package index to find the catalog leaf where your desired version resides.

If you know the version already, you can make a shortcut there, the protocol allows that.

Also, "recent.json" is a pwsh.gallery nomenclature. pwsh.gallery is optimized for the fact that the majority of powershell installations want the latest version and prerelease version, so to minimize bandwidth recent.json is generated automatically with only the dependency metadata included to minimize data transfer over the wire and serialization delay, but nuget v3 is designed with static fetching in mind.

From what I see the fetch is mostly working to spec for a cold fetch, though I don't think it caches or uses ETag today to speed up future queries. There's one call to recent.json that isn't necessary, since the index.json would show the version being searched for is in older.json

PowerShell / PSResourceGet