Bootstrapper doesn't work with some Nuget repos

bacar commented 8 years ago

Description

The bootstrapper does not work against our internal Nexus repo which is hosting Nuget instances. Having debugged the bootstrapper, it seems that this is because the Bootstrapper seems to exclusively use the /package-versions/ Nuget v2 API endpoint, which some Nuget repos don't seem to support, including our corporate Nexus installation (v2.13).

I'm unclear on whether it should support this because I can't find a clear documentation for either Nuget or Nexus... but I can see that paket.exe itself falls back to alternative OData query endpoints /FindPackagesById()?... and /Packages?$filter=... which both work fine, so Paket.exe doesn't have this problem - only the bootstrapper.

Repro steps

Difficult to specify beyond the description above as I cannot expose our corporate nexus repo externally.

I'm running

paket.bootstrapper.exe -f --force-nuget --nuget-source=http://my-nuget-repo/the/repo

Expected behavior

Bootstrapper downloads paket.exe

Actual behavior

Error printed to console. "The remote server turned an error: (404) Not Found. (Nuget)

Known workarounds

None.

Related information

Operating system = Windows 7 x64
Branch - v3.16.3.0

forki commented 8 years ago

Since you already debugged this - can you please create an fix?

Am 26.08.2016 12:52 vorm. schrieb "bacar" notifications@github.com:

Description

The bootstrapper does not work against our internal Nexus repo which is hosting Nuget instances. Having debugged the bootstrapper, it seems that this is because the Bootstrapper seems to exclusively use the /package-versions/ Nuget v2 API endpoint, which some Nuget repos don't seem to support, including our corporate Nexus installation (v2.13).

I'm unclear on whether it should support this because I can't find a clear documentation for either Nuget or Nexus... but I can see that paket.exe itself falls back to alternative OData query endpoints /FindPackagesById()?... and /Packages?$filter=... which both work fine, so Paket.exe doesn't have this problem - only the bootstrapper. Repro steps

Difficult to specify beyond the description above as I cannot expose our corporate nexus repo externally.

I'm running

paket.bootstrapper.exe -f --force-nuget --nuget-source=http://my-nuget-repo/the/repo

Expected behavior

Bootstrapper downloads paket.exe Actual behavior

Error printed to console. "The remote server turned an error: (404) Not Found. (Nuget) Known workarounds

None. Related information

Operating system = Windows 7 x64

Branch - v3.16.3.0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/1891, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNMlqg8PCDCfXprqypW2iyDSKrAPDks5qjhyugaJpZM4JtlTa .

bacar commented 8 years ago

Yeah I've been having a go already. Will let you know.

bacar commented 8 years ago

Have discovered some additional complexity. The bootstrapper assumes that the package is available from {repo-root}/package (latest) or {repo-root}/package/{version}, but again this appears to not be true for Nexus.

If you actually grab the feed such as https://www.nuget.org/api/v2/Packages?$filter=(Id%20eq%20%27Paket%27)&$orderby=Version%20desc&$top=1 (or equivalent), pulling out the content element seems to contain the correct URL.

<content type="application/zip" src="https://www.nuget.org/api/v2/package/Paket/3.9.7" />

I'm giving examples from nuget.org which obviously aren't going to show the problem with the internal repo, so sorry if that's causing any confusion - I just wanted to be clear which endpoints we're talking about. Obviously on our Nexus the content returns without the /package part of the path. Hey it's all HATEOAS-ey!

This is all based on observation - are there actually any NuGet web API docs anywhere?!

forki commented 8 years ago

are there actually any NuGet web API docs anywhere?!

;-) /cc @maartenba

maartenba commented 8 years ago

Nope. It's an OData feed (V2 at least), so go wild querying :-) (the proposed query could be a good fallback)

bacar commented 8 years ago

I should be able to put a PR together. I'm able to replicate the issue at home using an out-of-the box OSS Nexus from docker with a nuget.org proxy added.

I have been having some difficulty with the Nuget API.

Scraping all possible versions from Paket using the filter API and crawling the 'next' links until done seems to be surprisingly slow.
The Search() API seems faster but from what I can glean online the search term syntax for this is a Lucene search query. I'm loath enough to figure out Nuget's undocumented API, let alone rely on something that appear to be an internal implementation detail.
I've tried a few filters to speed things up - for example using $orderby=Version Desc and $top, but Version is a string and it sorts lexicographically, not by semantic version! Grrr. Could use Created (a DateTime) as a dubious proxy.

I finally found the following queries seem to give reasonable results: Latest release: [$filter=Id eq 'Paket' and IsLatestVersion eq true](http://www.nuget.org/api/v2/Packages?$filter=Id eq 'Paket' and IsLatestVersion eq true) Latest prerelease: [$filter=Id eq 'Paket' and IsAbsoluteLatestVersion eq true](http://www.nuget.org/api/v2/Packages?$filter=Id eq 'Paket' and IsAbsoluteLatestVersion eq true)

...but I haven't the faintest clue if I can rely on this... any guidance? I'll start with the last approach.

sideeffffect commented 8 years ago

@bacar there's this class SemVer which implements CompareTo, so maybe that could help.

@forki @maartenba is there Swagger description of NuGet.org's REST API?

fsprojects / Paket