Azure DevOps: NuGet constantly failing due to reliance on offline package cache

dasMulli commented 4 years ago

Details about Problem

dotnet.exe --version (if appropriate): 3.1.100

Worked before? If so, with which NuGet version:

Detailed repro steps so we can see the same problem

Create a project that references a lot of NuGet packages.
Create an Azure DevOps build definition fetching the 3.1.100 SDK and using dotnet build to restore and build the project.
Observe that builds fail occasionally. In our case, one project only succeeds in ~40% of cases.

Other suggested things

It looks like a Restore fails when a package cannot be found in the offline package source configured in Azure DevOps and the responses from nuget.org take a bit longer.

In our case, a solution with 21 projects with a total of 526 pakcage references fails to restore. There is no nuget.config or similar configuration as all packages are fetched from nuget.org.

For example, we see error messages in the build log that a package doesn't exist in the offline package source, but requests to nuget.org come back as OK:

However the overall build fails with NU1101 errors:

dasMulli commented 4 years ago

Apparently a workaround that seems to restore our pass rate so far is adding a nuget.config that clears all inherited feeds and adds nuget.org again. I don't think that this should not be necessary though..

rrelyea commented 4 years ago

Please work with AzureDevOps support if this problem happens by default. When you restore, at least one of the package feeds need to have the packages you need. Generally is it best practice to declare which feeds your build needs in a nuget.config in your repo.

Are you saying that having nuget.config in my repo like this work:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
</configuration>

but that this one doesn't work:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
</configuration>

dasMulli commented 4 years ago

Yes having the first one works, using none is prone to failure. I believe the "defaut" feeds in AzDO are very similar to what is present when installing VS..

I was wondering why NuGet complains about the packages not being in the offline package feed that is installed by VS.

I may have had this issue in my dev machine as well while i only had the VS 16.5p1 installed any only 3.0.0 packages in the offline package store, then used a non-preview sdk to restore 3.1.1 packages on a slow connection (train WiFi9), but haven't got a consistent repo.

So it looks like there may (!) be an issue if the nuget.org feed is slow (but available) for a large graph and the offline package store does not contain the packages to restore, NuGet will still error out with NU1101 even though the logs indicate that the packages in question have been downloaded successfully.

I should be able to privately provide a binlog of an occurrence

nkolev92 commented 4 years ago

Hey @dasMulli,

Were you able to confirm if this is a azure devops only issue? A new VS install usually creates the folder when it adds the config that points to that.

For reference, see docs about NuGet + sources available behavior.

https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#packagereference-and-sources

dasMulli commented 4 years ago

Yes it is AzDO only, none of the other developers working on impacted projects with non-preview tooling have had any problems.

Other projects were also running into it every few months - windows agents would just not be able to restore projects randomly for a day or a few days. I don't have current statistics though (i believe the last time this was severely impacting a project was in March or early April, which means i may no longer have binlogs available)

We moved as many projects off windows agents to ubuntu agents where possible (since this never occured on them), but some full framework projects are stuck on windows agents.

Again, the characteristics are:

More prone to happen on projects with larger graphs (>= 10 csproj, 20+ top level packages)
Can be fixed by adding a nuget.config that clears all feeds before adding the public nuget.org feed.
None of the affected projects use any private package source, only the public nuget.org source.
Happens intermittently for ~30-50% of builds for between a day and a week.
Triggering failing builds again will eventually allow them to succeed, resulting in a "flaky build" characteristic.

nkolev92 commented 4 years ago

Thanks for the details @dasMulli.

We'll try to understand the incidence of this problem on our end.

We also need to include some best practices instructors as far repo level nuget.configs go.

Tagging a few relevant folks.

@aortiz-msft @zkat @rconard @chgill-MSFT @JonDouglas

NuGet / Home