search

NuGet / Home

Repo for NuGet Client issues
Other
1.49k stars 250 forks source link

.NET SDK 8.0.401: random "multiple attempts to download the nupkg have failed" #13729

Open metoule opened 3 weeks ago

metoule commented 3 weeks ago

NuGet Product Used

dotnet.exe

Product Version

SDK Version: 8.0.401

Worked before?

8.0.304

Impact

It's more difficult to complete my work

Repro Steps & Context

Our CI randomly fails during the dotnet restore phase, with errors like the following:

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Grpc.Tools.2.65.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

Other random examples:

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Grpc.AspNetCore.Server.Reflection.2.65.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Dapper.2.1.35' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

Each time, the package that's not found is a different one, so it seems totally random.

Of course, all these packages exists:

Verbose Logs

No response

nicholasreinlein-broadsign commented 3 weeks ago

It seems completely random, and occurs with theses packages as well:

Reruns are successful

nkolev92 commented 3 weeks ago

Looks very much like: https://github.com/NuGet/Home/issues/13545.

This has been in the .NET 9 previews, maybe you can update to the latest .NET 9 previews to confirm whether you're still seeing issues?

Andrenhn commented 2 weeks ago

Will the fix be applied to .NET8 as well? Not that easy to just jump up to NET9.

nkolev92 commented 2 weeks ago

We haven't gotten these reports with .NET 8 SDKs much. There's no immediate plan to port it to .NET SDK 8.0.4xx, but that could change.

It'd be great if people can try out the .NET 9 SDK and help confirm that the root cause is the same. If you do that, please comment here.

andrstor commented 2 weeks ago

We have struggled with this for the last week or so. The problems began after the release of the latest .net 8 version. At first we just thought it was something with azure devops artifacts, but now we suspect that this is the bug we are seeing. The packages that fails are totally random, and it is hard to reproduce. It happens several times a day and is starting to become really annoying.

matoomx commented 2 weeks ago

We use the hosted agent ubuntu-latest (Current image version: '20240818.1.0') in our CI pipeline and we get this error on a daily basis. Right now we have to have an extra manual step in our process where failed pipline-runs are re-scheduled if we see this error in the log. I really hope this will be solved in .net 8 fix

Update: We have another pipeline that uses the windows-latest image (Current image version: '20240811.1.0') and that pipeline have never failed with this error and it uses .net core sdk 8.0.303

mabergstrom commented 2 weeks ago

I can just confirm that this issue has started to become really frustrating and happens several times during the day now. Seems to be completely random as people mentioned earlier.

WalissonPires commented 2 weeks ago

Today I came across the same issue in the pipeline on Azure DevOs when generating a docker image. Here the error was:

error : The feed '***.org [https://api.***.org/v3/index.json]' lists package 'Microsoft.EntityFrameworkCore.Design.8.0.4' but multiple attempts to download the nupkg have failed.

After running the pipeline again manually or restoring it ran successfully.

.NET SDK: 8.X

AzurePipeline: vmImage: ubuntu-latest task: DockerInstaller@0 inputs: dockerVersion: '20.10.9'

olaherrdahl commented 2 weeks ago

I just ran into this as well when testing the deploy-image workflow described here: https://docs.github.com/en/packages/managing-github-packages-using-github-actions-workflows/publishing-and-installing-a-package-with-github-actions#upgrading-a-workflow-that-accesses-a-registry-using-a-personal-access-token

#17 [build  9/12] RUN dotnet restore "./...csproj"
#17 1.063   Determining projects to restore...
#17 2.404 /usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Ninject.3.3.6' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [/src/....csproj]

No issues when running it manually a second time.

borland commented 2 weeks ago

Since .NET SDK 8.0.4xx has arrived, we've been seeing these NuGet download failure attempts intermittently too, wreaking havoc on our CI pipelines. It's good to see a fix is potentially in .NET 9, but that's an STS release which not everyone will be able to uptake. It'd be good to see a fix in 8.0.x as well please.

NicholasFields commented 2 weeks ago

TY@Niko for keeping us in the know.

CI has become gambling for my team as well.

Alongside @Andrenhn and @borland, We would very much appreciate consideration of a .NET8 patch.

Till then, good luck to all :)

iskiselev commented 2 weeks ago

We are affected by the issue for about last two weeks after updating to newer SDK on both Linux/Windows.

telb99 commented 2 weeks ago

We have started seeing this some time over the last two weeks also... feels like it is getting worse each day. This is on MS Hosted agents in ADO running ubuntu, and we are pulling all packages from a private package feed in ADO. We have seen this in docker builds also where we are using the 8.0-alpine image. Please port this back to 8.x, we are not able to jump to a preview version of .NET for our production applications.

BrettMillerDealX commented 1 week ago

We've been seeing this for over 2 weeks when running on BitBucket pipelines.

Multiple instances per day and different libraries each time all with same message.

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'runtime.any.System.Text.Encoding.4.3.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

It's been very disruptive.

Pucmaster commented 1 week ago

We have same problem in bitbucket pipelines after migration to net 8

sergei-lobanov commented 1 week ago

Same issue with .NET 8 again and again every day, its a blocker issue (!). We need a hotfix.

zivkan commented 1 week ago

We still think that https://github.com/NuGet/NuGet.Client/pull/5905 will fix the issue, and I'm in the process of trying to get approval for it. If we have customers report here saying they tested the .NET 9 preview SDK and the problem no longer occurs, then it gives the people who approve/reject servicing requests more confidence that we're fixing the right thing, and hopefully not risking new bugs.

It's good to see a fix is potentially in .NET 9, but that's an STS release which not everyone will be able to uptake.

Firstly, it was a request to temporarily test the .NET 9 SDK and provide feedback if it works. Not a request to permanently use the .NET 9 SDK.

Secondly, you can continue to target .NET 8 in your projects, when using the .NET 9 SDK. .NET 9 SDK are the build tools, not the runtime that projects being built has to use. It's no different to using Visual Studio 17.11 even though .NET 8 originally came out with Visual Studio 17.8, or using either of those versions of Visual Studio while running your apps on the .NET 6 LTS runtime. runtime version != SDK version.

Anyway, as mentioned, https://github.com/NuGet/NuGet.Client/pull/5905 is our best guess about the problems, so I'm working on trying to get it added. If it turns out that this won't fix the problem, then even when the .NET 8.0.4xx SDK is updated with a new version of NuGet, the problem might continue. Hence why we'd like customer validation that the problem is resolved in the preview version, to be confident that it's the right hotfix for the release version.

matoomx commented 1 week ago

I have installed .net 9 sdk with the following yaml in one of our pipelines and will run it as often as we have a free agent to see if the problem have dissapered. The installation only takes a couple of second so maybe the yaml is useful to someone else who wants to check if the problem is gone in .net 9 sdk

  - task: UseDotNet@2
    displayName: 'Install .NET 9 Core SDK'
    inputs:
      includePreviewVersions: true
      version: 9.x
      performMultiLevelLookup: true
matoomx commented 1 week ago

Now I have run a pipeline with .net 9 SDK 11 times and not seen this error once. For reference I have run a pipeline with.net 8 SDK 10 times and got this error 3 times, so something definitely seems better with .net 9 (9.0.100-preview.7.24407.12)

danielmarbach commented 1 week ago

Any chance the fix can get backported to 8.x to prevent this issues from happening there too given NET8 is an LTS version?

sergei-lobanov commented 1 week ago

same question... looking for a fix to be available in .NET8, we've tried to use .NET9 temporary, but looks like we can't use only as SDK for the build process in Azure pipeline, its failed with the request to upgrade the application to 9 version and we can't do it for production app.

Hope that confirmations above for v9 is enough to backport the fix to v8.

zivkan commented 1 week ago

but looks like we can't use only as SDK for the build process in Azure pipeline, its failed with the request to upgrade the application to 9

There's no such requirement built into the .NET SDK. You can absolutely target net8.0 while using the .NET 9 SDK.

matoomx commented 1 week ago

Is there anything else I/we can do to help in the process of getting this fix in a .net 8 release?

Although we can use .net 9 SDK in some azure pipelines and in others we have implemented nuget caching to manage, this bug is turning in to a horror in other scenarios. For example we build docker images where we use multistage builds with SDK matching runtime images and right now we have stopped this process because the build process fails to often with this error.

Right now our pipelines that use hosted agents for windows doesn’t get this problem because the current image we get is '20240811.1.0' and it uses SDK 8.0.303 but later builds of this agent will get 8.0.4x and when that happens this bug will affect many more pipelines and I guess many more customers.

vanavara-truecode commented 1 week ago

The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Autofac.Extensions.DependencyInjection.7.2.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again

The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Dapper.2.1.35' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again

The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Microsoft.EntityFrameworkCore.8.0.3' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again

The real chaos in our CI/CD. The errors above appear absolutely randomly. Manuall restarting pipeline can give random result for any dependencies.

Confirmed the reported issue.

What has happened? I'm affraid that migrating to .NET9 doesn't help at all.

BrettMillerDealX commented 3 days ago

Still seeing intermittent failures multiple time per day in our build pipelines using the LTS version of dotnet 8 docker files.

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

[build  3/10] RUN dotnet --info
sha256:62b6f21d7c15d0e1c6e9611a1449a81b0db6e502c68ff1e4105da7a14474826f
0.650 .NET SDK:
0.650  Version:           8.0.401
0.650  Commit:            811edcc344
0.693  Workload version:  8.0.400-manifests.b6724b7a
0.695  MSBuild version:   17.11.4+37eb419ad
0.695 
0.696 Runtime Environment:
0.697  OS Name:     debian
0.697  OS Version:  12
0.703  OS Platform: Linux
0.718  RID:         linux-x64
0.718  Base Path:   /usr/share/dotnet/sdk/8.0.401/
0.719 
0.719 .NET workloads installed:
0.723 Configured to use loose manifests when installing new manifests.
0.723 There are no installed workloads to display.
0.724 
0.724 Host:
0.724   Version:      8.0.8
0.724   Architecture: x64
0.724   Commit:       08338fcaa5
0.724 
0.724 .NET SDKs installed:
0.724   8.0.401 [/usr/share/dotnet/sdk]
0.724 
0.724 .NET runtimes installed:
0.724   Microsoft.AspNetCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
0.724   Microsoft.NETCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

-- snip -- 

[build  9/10] RUN dotnet restore
sha256:5ab46877abc47c6a61b2114728a2455206be91871a5d36fd16fb4334542ef3d7
1.517   Determining projects to restore...
3.878 /usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Dapper.2.1.35' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. 
4.222   Failed to restore *snip* (in 1.03 sec).
5.819 /usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Dapper.2.1.35' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. 
ERROR: process "/bin/sh -c dotnet restore" did not complete successfully: exit code: 1
psergei commented 2 days ago

Same here, pipeline in bitbucket fails multiple times everyday. Using .NET8

emreyigit commented 2 days ago

Same here for System.Linq.Async package while using .NET8.

metoule commented 2 days ago

As the original author of this issue, I get pinged on every comment. Can you please refrain from commenting "I have the same issue"? I sympathize with your pain, but every time I'm hoping for an update from the NuGet team, and every time I'm disappointed. Thank you :)