Open xsoheilalizadeh opened 3 years ago
Facing the same issue with .NET 5
Adding --disable-parallel fixed my issue but it takes a long time to restore.
Any updates on this ?
I'm also facing the same issue. My project is built on top of .NET Core 3.1 LTS.
Any updates?
+1
this happens intermittently within docker for me
Same here, with latest aspnet:3.1.10-bionic
--disable-parallel
helps but can take minutes to restore
Using internal repo instead of api.nuget.org gives the same results
+1
Looks like throttling the number of requests with nuget.config helps and works much faster than 'disable-parallel' (dont forget to copy it into container)
<configuration>
<config>
<add key='maxHttpRequestsPerSource' value='20' />
</config>
...
This issue impacts my daily work because I have to wait about ~4 min for each package restore on the container, could anyone in the NuGet team have (just) a look at this issue help us to fix it, it becomes frustrating.
I also tried to mount the NuGet cache folder from host to container but it still takes a lot of time to be restored.
volumes:
- ~/.nuget/packages:/root/.nuget/packages:rw
@JonDouglas, @nkolev92, @karann-msft
cc @zivkan
@xsoheilalizadeh Can you give us some details about your setup?
Is the host and containers Windows or Linux or Mac? When you restore on the host does it have the same problem, or only in contains? Are you limiting the container resources (ram and/or cpu), or does it have the same resources as the host?
Is the host and containers Windows or Linux or Mac?
I used both Linux and Windows on the same machine as a Host.
When you restore on the host does it have the same problem, or only in contains?
The host is able to restore perfectly without any issues, either it's from cache or NuGet server.
Are you limiting the container resources (ram and/or cpu), or does it have the same resources as the host?
No resource limitation has been set, everything is the default.
I tried your repro (single project, small number of packages), as well as the NuGet.Client repo (large number of projects and packages). I tried on my computer with both Windows and Linux contains, and I tried on 2-core Azure VMs using Ubuntu 20.10 as the host, and once I noticed that you were using Debian 9, I also tried Debian 9 as a host. In all of these attempts, I was not able to reproduce the issue. Restore worked inside the container in all my tests.
I never used Docker before this week, so if anyone has any suggestions on how I might repro the issue, please let me know.
Having the same issue on Mac with .NET 5 Docker 20.10. Any update on this? --disable-parallel works but takes 10x longer
Just to complete the working answer by Vfialkin
putting a file name nuget.config
(watch the casing) besids the solution works.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<config>
<add key='maxHttpRequestsPerSource' value='10' />
</config>
</configuration>
So maybe the issue needs to be fixed by microsoft by changing the default nuget.config
inside of mcr.microsoft.com/dotnet/sdk:5.0
. So the issue is this one
https://github.com/dotnet/dotnet-docker/issues/1670
Looks like throttling the number of requests with nuget.config helps and works much faster than 'disable-parallel' (dont forget to copy it into container)
<configuration> <config> <add key='maxHttpRequestsPerSource' value='20' /> </config> ...
Think I hit the same issue here where nuget just will not download the packages from a dev ops artifact feed and throws that time out error. Running dotnet restore from a ACI self hosted dev ops agent with mcr.microsoft.com/dotnet/sdk:3.1-focal
as the base image
Ran into this issue few days ago, blocking all our Docker pipelines when trying to restore from a private Artifactory.
Tried with dotnet restore --configfile NuGet.config
, with:
<add key="maxHttpRequestsPerSource" value="10" />
as NuGet configuration (first with 25, then 15, 10 and even 5).
Also tried a retry system that fallbacks to the same command with --disable-parallel
:
RUN if !(dotnet restore ${ATHENA_MODULE_PATH}.sln --configfile NuGet.config --disable-parallel); then \
echo "WARNING: dependency restore failed, retrying without parallelization" && \
for retry in $(seq 1 5); do dotnet restore ${ATHENA_MODULE_PATH}.sln --configfile NuGet.config --disable-parallel && result=0 && break || \
result=$? && echo "WARNING: non-parallelized dependency restore failed, retrying in 30 seconds (${retry}/5)" && sleep 30; done; (exit $result); fi
(quite ugly I know, but this was the best idea I could come up with late at night). Whatever the fix I chose, the restore ultimately failed given enough time.
Until, in a last desperate attempt, I changed the base Docker image from:
mcr.microsoft.com/dotnet/sdk:3.1-focal
to
mcr.microsoft.com/dotnet/sdk:3.1-bullseye
Seems too good to be true, but this fixed everything.
I haven't had any issues since.
EDIT: well, my apologies, the "Debian" fix only lasted a couple of days (still, impressive).
However, I did find this:
https://www.jfrog.com/jira/si/jira.issueviews:issue-html/RTFACT-20557/RTFACT-20557.html
TL;DR: if your private NuGet source connects to nuget.org to fetch its packages, there's an issue where nuget.org sometimes times out and returns a 404 Not Found
error to the connected source, instead of pulling the cache (or sending a more meaningful error). This affects the latest Artifactory release (and many before).
In my case, it explains a lot, and I hope it will for some of you too.
Any updates on this one, it is persisting on .net 6 as well
@nkolev92 @aortiz-msft Unassigned myself for load balancing.
This happens to me as well. it sometimes works but fails the builds most of the time.
@Talhafayyaz11 I'm trying to repro the issue on Docker for Windows. Can you provide the exact steps you're doing that experienced the problem? Thanks in advance!
Also having this issue, we are running in docker with the sdk:6.0 image , host is on aws code build amazonlinux2-x86_64-standard:3.0 image
@cli00004 are you able to repro the problem consistently? I am still unable to reproduce it in order to try and narrow down a cause
hi @jeffkl i'm able to reproduce it consistently on every try, haven't got a successful build yet
@cli00004 can you please provide exact steps I can follow and I can see if I can get it to happen? Thanks in advance!
Sure @jeffkl
This is a simplified version of the dockerfile we use
This is the error i'm getting
The authentication seems working it's able to get the index file, just getting timeout trying to restore
@jeffkl I think this might be an issue with ADO Artifacts, i tried pushing the package to Github package, it works fine there
I spent about three days investigating this and trying to repro the issue but am not able to. I tried the following:
If its related to thread starvation which is what some has suggested, it would be very much appreciated if people could provide more details about their environment (CPU cores) and exact repro steps I can follow. I really want to get this figured out!
I had this issue on my old machine which had 4 CPU cores.
I've spent a lot of time trying to reproduce this issue but have not been able to. Is someone able to provide precise steps to reproduce the issue?
@jeffkl I think that slow internet connection is the key in order to reproduce it. I have an i7 6700HQ mobile with 16gb ram, and without setting max parallel option it does not work.I had tried 2 years ago from a high speed ether net connection with the same laptop, without issues. With this option set, it takes a long time to restore a big solution.
Any updates on this one, it is persisting on .net 5 as well
@yusufgungor can you post precise steps I can follow to reproduce the issue? I have tried to get it to happen so I can figure out what's going on but can't seem to.
Hello,
Anyone have luck with this so far ?
Here is how I am able to reproduce this:
failed at restore
and fails to resolve dns
running ping command against ip address works OK
@janivimal Are you able to reproduce this over and over on the same machine? I just did it with this sample on my local Docker Desktop and its working.
What does tracert
return? Its strange that you can't resolve the api.nuget.org
host, do you have any custom DNS configuration?
tracert api.nuget.org
Tracing route to cs2.wpc.gammacdn.net [152.199.4.184]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.1.1
2 16 ms 13 ms 14 ms 96.120.101.197
3 12 ms 11 ms 12 ms 96.110.249.37
4 13 ms 12 ms 12 ms po-2-rur102.bellevue.wa.seattle.comcast.net [69.139.160.26]
5 17 ms 11 ms 11 ms po-100-xar02.bellevue.wa.seattle.comcast.net [69.139.164.77]
6 14 ms 14 ms 14 ms 24.124.128.89
7 13 ms 13 ms 14 ms be-36131-cs03.seattle.wa.ibone.comcast.net [68.86.93.9]
8 14 ms 15 ms 14 ms be-2301-pe01.seattle.wa.ibone.comcast.net [96.110.39.226]
9 16 ms 14 ms 15 ms 96-87-9-30-static.hfc.comcastbusiness.net [96.87.9.30]
10 20 ms 16 ms 16 ms ae-65.core1.sec.edgecastcdn.net [152.195.92.129]
11 14 ms 12 ms 15 ms 152.199.4.184
Trace complete.
@jeffkl , Are you able to reproduce this over and over on the same machine? - Yes, I am able to reproduce this over and over on the same machine Also, tried running tracert and says "unable to resolve target system name api.nuget.org."
Next, I will try building these samples and share result with you
Tried building image from these samples (dotnetapp) and same error:
@janivimal I'm thinking you're facing a different issue where your docker VM is unable to resolve DNS entries. Without that working, you won't be able to restore packages. Do you know why DNS isn't working?
@jeffkl correct , something is blocking my containers to restore packages. Again this issue only occurs when I am building windows containers. It runs perfectly fine when I switch docker to Linux Container mode and build appropriate image.
Trying to play around with installing certs in container before restore command gets executed.
Windows containers on Docker Desktop use Hyper-V as far as I know. Linux containers on Docker Desktop can use WSL or Hyper-V. I can see how on a Windows machine, you could get different network results for the different containers. Can you see if your Hyper-V configuration is preventing DNS from working? Perhaps the VM is only connected to the private VLAN and not the public internet?
+1
@jeffkl could you please provide more details on how to achieve - "Can you see if your Hyper-V configuration is preventing DNS from working?" ,
Can someone please try to reproduce this ? Below are the steps, that I am following
Create .net 6.0 api using VS2022 - selecting windows container when setting up project (windows machine if possible)
trim down docker file to below FROM mcr.microsoft.com/dotnet/sdk:6.0 WORKDIR /src COPY ["{your project}.csproj", ""] RUN dotnet restore "./{your project}.csproj" --verbosity detailed
validate and switch docker desktop to Windows Container mode
from PS command prompt , navigate to csproj level, run below command
docker build .
Thank you !
@jeffkl -
Thank you very much for helping me resolve my issue.
Finally this was resolved. Issue was my ANTIVIRUS , which was messing up internal connectivity between my container and host (not sure exactly why). As soon I un-installed my antivirus and tried rebuilding my image it worked !!!!
Last few days of frustration finally ended.
Thanks a million !
Forwarded form dotnet/dotnet-docker#1670
I had a problem when building a sample dotnet app When I run
dotnet restore
in docker image I get following errors:Failed to download package 'Microsoft.AspNetCore.Authentication.Core.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.core/2.2.0/microsoft.aspnetcore.authentication.core.2.2.0.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.core/2.2.0/microsoft.aspnetcore.authentication.core.2.2.0.nupkg' timed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Failed to download package 'Microsoft.Extensions.Caching.Abstractions.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.caching.abstractions/2.2.0/microsoft.extensions.caching.abstractions.2.2.0.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.caching.abstractions/2.2.0/microsoft.extensions.caching.abstractions.2.2.0.nupkg' timed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Failed to download package 'Microsoft.AspNetCore.Diagnostics.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.diagnostics/2.2.0/microsoft.aspnetcore.diagnostics.2.2.0.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.diagnostics/2.2.0/microsoft.aspnetcore.diagnostics.2.2.0.nupkg' timed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Failed to download package 'Microsoft.Extensions.Logging.Abstractions.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.abstractions/2.2.0/microsoft.extensions.logging.abstractions.2.2.0.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.abstractions/2.2.0/microsoft.extensions.logging.abstractions.2.2.0.nupkg' timed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Failed to download package 'System.Net.WebSockets.WebSocketProtocol.4.5.3' from 'https://api.nuget.org/v3-flatcontainer/system.net.websockets.websocketprotocol/4.5.3/system.net.websockets.websocketprotocol.4.5.3.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/system.net.websockets.websocketprotocol/4.5.3/system.net.websockets.websocketprotocol.4.5.3.nupkg' timed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Failed to download package 'Microsoft.AspNetCore.Localization.Routing.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.localization.routing/2.2.0/microsoft.aspnetcore.localization.routing.2.2.0.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.localization.routing/2.2.0/microsoft.aspnetcore.localization.routing.2.2.0.nupkg' timed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Failed to download package 'Microsoft.EntityFrameworkCore.Tools.2.2.4' from 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.tools/2.2.4/microsoft.entityframeworkcore.tools.2.2.4.nupkg'. The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.tools/2.2.4/microsoft.entityframeworkcore.tools.2.2.4.nupkg' timed out because no data was received for 60000ms. ...
Dockerfile
# https://hub.docker.com/_/microsoft-dotnet-core FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build WORKDIR /source # copy csproj and restore as distinct layers COPY *.csproj . RUN dotnet restore # copy and publish app and libraries COPY . . RUN dotnet publish -c release -o /app --no-restore # final stage/image FROM mcr.microsoft.com/dotnet/core/runtime:3.1 WORKDIR /app COPY --from=build /app . ENTRYPOINT ["dotnet", "dotnetapp.dll"]
File *.csproj
<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>netcoreapp3.1</TargetFramework> </PropertyGroup> <ItemGroup> <PackageReference Include="Ben.BlockingDetector" Version="0.0.3" /> <PackageReference Include="Microsoft.AspNetCore.App" Version="2.2.7" /> <PackageReference Include="MongoDB.Driver" Version="2.9.2" /> <PackageReference Include="prometheus-net" Version="3.1.4" /> <PackageReference Include="prometheus-net.AspNetCore" Version="3.1.4" /> <PackageReference Include="Quartz" Version="3.0.7" /> <PackageReference Include="ServiceStack.Core" Version="5.7.1" /> <PackageReference Include="ServiceStack.OrmLite.MySqlConnector.Core" Version="5.7.1" /> <PackageReference Include="ServiceStack.OrmLite.PostgreSQL.Core" Version="5.7.1" /> <PackageReference Include="ServiceStack.Redis.Core" Version="5.7.1" /> <PackageReference Include="System.IO.Pipelines" Version="4.6.0" /> </ItemGroup> </Project>
Output of docker version
Client: Docker Engine - Community Version: 19.03.2 API version: 1.40 Go version: go1.12.8 Git commit: 6a30dfca03 Built: Thu Aug 29 05:29:49 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.2 API version: 1.40 (minimum version 1.12) Go version: go1.12.8 Git commit: 6a30dfca03 Built: Thu Aug 29 05:28:23 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.6 GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc: Version: 1.0.0-rc8 GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f docker-init: Version: 0.18.0 GitCommit: fec3683
Output of docker info
Client: Debug Mode: false Server: Containers: 46 Running: 6 Paused: 0 Stopped: 40 Images: 1338 Server Version: 19.03.2 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 4.9.0-4-amd64 Operating System: Debian GNU/Linux 9 (stretch) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.829GiB Name: debian-167 ID: K5NH:LCRJ:AFV7:D3TN:N3IN:KE6B:RICT:7SPD:A3HP:IPGW:OOIN:6XFQ Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support
\cc @JonDouglas
I think you should delete the .zip file before docker-compose up -d --build
I am also affected and would like to fix this. Pure linux docker containers running on linux
Almost 2 years later and still not resolved.
.NET6
Almost 2 years later and still not resolved.
.NET6
Do you have repro steps? Is it consistent? Everything we need to memory dump or repro steps, so far it didn't happen when we try to diagnose it.
Any update? please!
Forwarded form https://github.com/dotnet/dotnet-docker/issues/1670
I had a problem when building a sample dotnet app When I run
dotnet restore
in docker image I get following errors:Dockerfile
File *.csproj
Output of docker version
Output of docker info
\cc @JonDouglas