Package resolution randomly fails with Artifactory 5.4.6

[Katulus]

Description

We use Artifactory 5.4.6 as package server and we use Paket as package manager for our projects. Since we upgraded some of our projects to .NET Core 2 we upgraded Paket from 4.x to 5.91.0 becuase support of .NET Core 2 (netstandard2.0, netcoreapp2.0, ...) was added somewhere in 5.x. Since then the builds randomly fail when resolving packages from Artifactory.

Repro steps

Repro steps are quite vague but all I can say is that combination of Paket and Artifactory of certain versions has issues.

1. Have Artifactory 5.4.6 installed as package server

2. Use Paket 5.91.0 to resolve packages from Artifactory

Expected behavior

Packages are resolved, installed, everything works.

Actual behavior

Package resolution gets lots of warnings and randomly fails or times out because it's not able to find proper packages. But then the next run usually passes, however still with warnings in log.

Examples of errors:

[08:42:53][Step 1/1] At least one 'next' link (index 0) returned a empty result (noticed on 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget_libs/FindPackagesById()?semVerLevel=2.0.0&id='docfx.console'&$orderby=Published desc'): ['http://dev-artifactory.dev.local:80/artifactory/api/nuget/nuget_libs/FindPackagesById()?$skip=80&semVerLevel=2.0.0&$orderby=Published desc&id='docfx.console'']
[08:42:53][Step 1/1]  - docfx.console 2.24.0
[08:42:55][Step 1/1] At least one 'next' link (index 0) returned a empty result (noticed on 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget_libs/FindPackagesById()?semVerLevel=2.0.0&id='vswhere'&$orderby=Published desc'): ['http://dev-artifactory.dev.local:80/artifactory/api/nuget/nuget_libs/FindPackagesById()?$skip=80&semVerLevel=2.0.0&$orderby=Published desc&id='vswhere'']
[08:42:55][Step 1/1] At least one 'next' link (index 0) returned a empty result (noticed on 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget_libs/FindPackagesById()?semVerLevel=2.0.0&id='Pickles.CommandLine'&$orderby=Published desc'): ['http://dev-artifactory.dev.local:80/artifactory/api/nuget/nuget_libs/FindPackagesById()?$skip=80&semVerLevel=2.0.0&$orderby=Published desc&id='Pickles.CommandLine'']
[08:42:55][Step 1/1]  - Pickles.CommandLine 2.16.2
[08:42:55][Step 1/1] Source 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget-OrionPlatform-stable' exception: System.Exception: error ---> System.Exception: Could not retrieve data from 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget-OrionPlatform-stable/Packages(Id='vswhere',Version='2.1.3')' ---> Paket.Utils+RequestFailedException: Request to 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget-OrionPlatform-stable/Packages(Id='vswhere',Version='2.1.3')' failed with: 'NotFound'
[08:42:55][Step 1/1]    at Paket.Utils.failIfNoSuccess@498-5.Invoke(RequestFailedInfo _arg1)
[08:42:55][Step 1/1]    at Microsoft.FSharp.Control.AsyncBuilderImpl.args@823-1.Invoke(a a)
[08:42:55][Step 1/1]    --- End of inner exception stack trace ---
[08:42:55][Step 1/1]    at Paket.Utils.getFromUrl@694-8.Invoke(Exception _arg4)
[08:42:55][Step 1/1]    at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@839.Invoke(AsyncParams`1 args)
[08:42:55][Step 1/1]    --- End of inner exception stack trace ---
[08:42:55][Step 1/1]    at Paket.NuGetV2.getDetailsFromNuGetViaODataFast@373-1.Invoke(FSharpChoice`2 _arg5)
[08:42:55][Step 1/1]    at Microsoft.FSharp.Control.AsyncBuilderImpl.args@823-1.Invoke(a a)
[08:42:55][Step 1/1] Source 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget-OrionPlatform-stable' exception: System.Exception: error ---> System.Exception: Could not retrieve data from 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget-OrionPlatform-stable/Packages(Id='Pickles.CommandLine',Version='2.16.1')' ---> Paket.Utils+RequestFailedException: Request to 'http://dev-artifactory.dev.local/artifactory/api/nuget/nuget-OrionPlatform-stable/Packages(Id='Pickles.CommandLine',Version='2.16.1')' failed with: 'NotFound'
[08:42:55][Step 1/1]    at Paket.Utils.failIfNoSuccess@498-5.Invoke(RequestFailedInfo _arg1)
[08:42:55][Step 1/1]    at Microsoft.FSharp.Control.AsyncBuilderImpl.args@823-1.Invoke(a a)
[08:42:55][Step 1/1]    --- End of inner exception stack trace ---
[08:42:55][Step 1/1]    at Paket.Utils.getFromUrl@694-8.Invoke(Exception _arg4)
[08:42:55][Step 1/1]    at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@839.Invoke(AsyncParams`1 args)
[08:42:55][Step 1/1]    --- End of inner exception stack trace ---
[08:42:55][Step 1/1]    at Paket.NuGetV2.getDetailsFromNuGetViaODataFast@373-1.Invoke(FSharpChoice`2 _arg5)
[08:42:55][Step 1/1]    at Microsoft.FSharp.Control.AsyncBuilderImpl.args@823-1.Invoke(a a)

All the packages are present on the server with correct versions. If I re-run the build it usually passes with exactly the same packages available.

I assume that main problem is no support of SemVer 2 in Artifactory and they have a case for that already - https://www.jfrog.com/jira/browse/RTFACT-13090. But if Paket is not able to work without SemVer 2 (even though it used to in 4.x branch) then it's a blocker for us being on .NET Core 2.0.

Known workarounds

Switch to Nuget. :-(

[forki]

/cc @matthid

[matthid]

Did you try with latest as well?

[Katulus]

Yes, I did. I have not applied it to our build configurations globally to see if it has better "success rate" but I tried it and still received some failed builds and same errors in build log.

[Katulus]

I don't think that root of the issue is in Paket. Lack of SemVer 2 support in Artifactory is probably the main cause. But because it was working fine in 4.x version (specifically 4.1.5 which is used through our company as default) it would be great to be able to get the same behavior with 5.x as well. Even something like --legacy switch would work.

[matthid]

So if I understand that correctly it breaks because we add semVerLevel=2.0.0?

[cdrnet]

This is likely also related to the feed paging which is still broken in artifactory (bad next links). There was a rather poor but effective work around implemented in #2073, which unfortunately has since regressed due to lack of integration tests (in turn due to lack of a way to test against an artifactory server, see #2511).

[forki]

if someone could set up a repro - we could probably fix this soonish

[Katulus]

So if I understand that correctly it breaks because we add semVerLevel=2.0.0?

It does not look like the cause. I compared responses from Artifactory for both with semVerLevel and without and they are the same.