filter `IsLatestVersion` returns invalid response for package `Microsoft.PowerShell.PSResourceGet`

sean-r-williams commented 6 months ago

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest version
[X] Search the existing issues.

Steps to reproduce

Call FindPackagesById() for package JiraPS via the PSGallery NuGet v2 API - example API call: https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27JiraPS%27&$filter=IsLatestVersion
Repeat the same API call for package Microsoft.PowerShell.PSResourceGet - example call: https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27Microsoft.PowerShell.PSResourceGet%27&$filter=IsLatestVersion
Repeat the same API call for package Microsoft.PowerShell.PSResourceGet, but this time add a space to the front of the filter string: https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27Microsoft.PowerShell.PSResourceGet%27&$filter=%20IsLatestVersion

Expected behavior

All 3 API calls return one and only one package (with the latest version) for their respective package IDs

Actual behavior

API call #2 returns invalid NuGet XML response, but calls 1 and 3 work as expected

Error details

N/A

Environment data

N/A

Version

N/A

Visuals

XML response from API call number 2 is:

<?xml version="1.0" encoding="utf-8"?>
<feed xml:base="https://www.powershellgallery.com/api/v2"
      xmlns="http://www.w3.org/2005/Atom"
      xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
      xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
      xmlns:georss="http://www.georss.org/georss"
      xmlns:gml="http://www.opengis.net/gml">
    <m:count>1</m:count>
    <id>http://schemas.datacontract.org/2004/07/</id>
    <title/>
    <updated>2024-05-15T01:39:20Z</updated>
    <link rel="self"
          href="https://www.powershellgallery.com/api/v2/Packages"/>
    <author>
        <name/>
    </author>
</feed>

Of particular interest is:

<m:count>1</m:count> suggests one <entry> should be present, but there are none.
<author> should be under an <entry>, not the <feed>. (It's also null for some reason?)

sean-r-williams commented 6 months ago

It should be noted that this blocks anyone using JFrog Artifactory from installing (or updating) PSResourceGet.

The misbehavior only occurring when the filter string is exactly IsLatestVersion suggests PSGallery may be attempting to do some custom behavior (based on raw filter-string matching), but that custom behavior is malfunctioning.

sean-r-williams commented 6 months ago

Adding a touch more context (I thought I had posted this elsewhere, but maybe not) - we only started seeing this behavior after PSResourceGet v1.0.4.1 was released. All other versions before/after this release (including prereleases) were SemVer-compliant, but 4-number versions are not.

My suspicion is that whatever Gallery is doing with the filter IsLatestVersion doesn't handle the case of a package ID offering both SemVer and non-SemVer versions.

cmkb3 commented 6 months ago

https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27Microsoft.Powershell.PSResourceGet%27&$filter=IsLatestVersion%20eq%20true appears to return the correct result, but as stated elsewhere $filter=IsLatestVersion should be a unary comparison.

sean-r-williams commented 6 months ago

https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27Microsoft.Powershell.PSResourceGet%27&$filter=IsLatestVersion%20eq%20true appears to return the correct result, but as stated elsewhere $filter=IsLatestVersion should be a unary comparison.

I think this falls into the case where any filter-string that is not the exact string IsLatestVersion works as expected. The difference between examples 2 and 3 show similar behavior.

sean-r-williams commented 6 months ago

@alerickson @SydneyhSmith Could someone from your team take a look at what's happening with this package? The default package manager being unable to update/install itself from the default package management service (albeit through a mirror, however with parameters the package manager has specified) is a pretty frustrating blocker to our internal rollout of PSRG.

Our plan was to provide devices with a bootstrap copy of PSResourceGet over the network, then use that bootstrap copy to fetch the latest version via PSGallery (through Artifactory, as this is an isolated network). The aforementioned bug with Gallery blocks us from doing this.

SydneyhSmith commented 6 months ago

@sean-r-williams apologies for the delay in response on this-- had some high pri things come up this week and I was away at Build-- have this flagged for the team to review at triage on Tuesday

anamnavi commented 6 months ago

@sean-r-williams thanks for the detailed issue and suggestions. We're going to look into the Gallery codebase to see what is happening and if we can't safely make the change there then we'll open a fix client side. Thanks for also checking @cmkb3

sean-r-williams commented 5 months ago

@anamnavi @SydneyhSmith has your team been able to make any headway here? It's been over a week and I haven't seen any changes to the status page on this repo or in the behavior of Gallery itself.

alerickson commented 5 months ago

@sean-r-williams we took a look into this into and it's a bit tricky. We initially thought things were broken for Find-PSResource as well, but that doesn't seem to be the case. Can I ask what you're using these request calls for? Maybe we can find a workaround or way to accommodate for your scenario?

alerickson commented 5 months ago

@sean-r-williams this issue seems to be a very niche issue that happens when a stable release of a lower version is published after the preview another. We tested a few scenarios and this is the only scenario where we were able to repro the issue. It seems like this issue has existed for a long time since there haven't been any changes made to sections that touch this code in many, many years.

sean-r-williams commented 5 months ago

@alerickson We are indeed using Find-PSResource (see PowerShell/PSResourceGet#1656).

The key scenario-wise difference is that we're proxying PSGallery through Artifactory (via a remote-proxy repo) as this is an isolated environment without Internet access for client machines. PSResourceGet talks to Artifactory, which then talks to Gallery. Artifactory transparently passes params on calls like FindPackageById() to the upstream, so Gallery will receive params that more closely match what Artifactory receives.

For example, when PSRG directly queries Gallery for the PSRG package, the API call looks something like this:

https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27Microsoft.PowerShell.PSResourceGet%27&$inlinecount=allpages&$filter=IsLatestVersion%20and%20Id%20eq%20%27Microsoft.PowerShell.PSResourceGet%27

When PSRG queries Artifactory, the call looks like this:

https://artifactory.f.q.d.n/artifactory/api/nuget/v2/psgallery-nuget-remote/FindPackagesById()?id=%27Microsoft.PowerShell.PSResourceGet%27&$inlinecount=allpages&$filter=IsLatestVersion

Artifactory, when forwarding that call upstream, makes the following request:

https://www.powershellgallery.com/api/v2/FindPackagesById()?id=%27Microsoft.PowerShell.PSResourceGet%27&$inlinecount=allpages&$filter=IsLatestVersion

You can see how the $filter params differ between the first and third URLs.

You mention that there hasn't been changes to this part of the Gallery codebase in many years - did PowerShellGet have any sort of provider-specific configuration/quirk flags like PSResourceGet does? Is it possible that the introduction of quirk flags in PSResourceGet made this issue more readily-apparent to Gallery (and PSResourceGet) users?

JFrog made it pretty clear that the transparent proxying of requests to Gallery was intentional behavior (and I agree with them - there's no way they could comprehensively solve this for every NuGet workflow) so it's not clear what the next steps are. Is this something resolvable within Gallery, or does a fix need to be shipped to PSResourceGet for the scenario of Gallery-by-proxy?

SydneyhSmith commented 5 months ago

Thanks @sean-r-williams we are going to try a client side fix for this and plan to pass along the build for you to test

sean-r-williams commented 4 months ago

@alerickson @SydneyhSmith FWIW I can now repro this with other packages as well, such as JiraPS (API call [here](https://www.powershellgallery.com/api/v2/FindPackagesById()?$filter=IsLatestVersion&$inlinecount=allpages&id=%27JiraPS%27))

JiraPS didn't perform the same stable->prerelease sequence PSResourceGet did. All that occurred recently was the prerelease of a higher version - no stable release. This API call was working prior to them minting a prerelease. Once again, munging the $filter parameter to literally any other string than IsLatestVersion fixes the problem.

This seems to be a far easier case to hit. Can your team confirm whether this is the same issue?

PowerShell / PowerShellGallery