NuGet failed to restore packages inside docker

xsoheilalizadeh commented 3 years ago

Forwarded form https://github.com/dotnet/dotnet-docker/issues/1670

I had a problem when building a sample dotnet app When I run dotnet restore in docker image I get following errors:

Failed to download package 'Microsoft.AspNetCore.Authentication.Core.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.core/2.2.0/microsoft.aspnetcore.authentication.core.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.core/2.2.0/microsoft.aspnetcore.authentication.core.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.Extensions.Caching.Abstractions.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.caching.abstractions/2.2.0/microsoft.extensions.caching.abstractions.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.caching.abstractions/2.2.0/microsoft.extensions.caching.abstractions.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.AspNetCore.Diagnostics.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.diagnostics/2.2.0/microsoft.aspnetcore.diagnostics.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.diagnostics/2.2.0/microsoft.aspnetcore.diagnostics.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.Extensions.Logging.Abstractions.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.abstractions/2.2.0/microsoft.extensions.logging.abstractions.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.abstractions/2.2.0/microsoft.extensions.logging.abstractions.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'System.Net.WebSockets.WebSocketProtocol.4.5.3' from 'https://api.nuget.org/v3-flatcontainer/system.net.websockets.websocketprotocol/4.5.3/system.net.websockets.websocketprotocol.4.5.3.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/system.net.websockets.websocketprotocol/4.5.3/system.net.websockets.websocketprotocol.4.5.3.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.AspNetCore.Localization.Routing.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.localization.routing/2.2.0/microsoft.aspnetcore.localization.routing.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.localization.routing/2.2.0/microsoft.aspnetcore.localization.routing.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.EntityFrameworkCore.Tools.2.2.4' from 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.tools/2.2.4/microsoft.entityframeworkcore.tools.2.2.4.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.tools/2.2.4/microsoft.entityframeworkcore.tools.2.2.4.nupkg' timed out because no data was received for 60000ms.
...

Dockerfile

# https://hub.docker.com/_/microsoft-dotnet-core
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /source

# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore

# copy and publish app and libraries
COPY . .
RUN dotnet publish -c release -o /app --no-restore

# final stage/image
FROM mcr.microsoft.com/dotnet/core/runtime:3.1
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

File *.csproj

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Ben.BlockingDetector" Version="0.0.3" />
    <PackageReference Include="Microsoft.AspNetCore.App" Version="2.2.7" />
    <PackageReference Include="MongoDB.Driver" Version="2.9.2" />
    <PackageReference Include="prometheus-net" Version="3.1.4" />
    <PackageReference Include="prometheus-net.AspNetCore" Version="3.1.4" />
    <PackageReference Include="Quartz" Version="3.0.7" />
    <PackageReference Include="ServiceStack.Core" Version="5.7.1" />
    <PackageReference Include="ServiceStack.OrmLite.MySqlConnector.Core" Version="5.7.1" />
    <PackageReference Include="ServiceStack.OrmLite.PostgreSQL.Core" Version="5.7.1" />
    <PackageReference Include="ServiceStack.Redis.Core" Version="5.7.1" />
    <PackageReference Include="System.IO.Pipelines" Version="4.6.0" />
  </ItemGroup>
</Project>

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfca03
 Built:             Thu Aug 29 05:29:49 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfca03
  Built:            Thu Aug 29 05:28:23 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info

Client:
 Debug Mode: false

Server:
 Containers: 46
  Running: 6
  Paused: 0
  Stopped: 40
 Images: 1338
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.0-4-amd64
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.829GiB
 Name: debian-167
 ID: K5NH:LCRJ:AFV7:D3TN:N3IN:KE6B:RICT:7SPD:A3HP:IPGW:OOIN:6XFQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

\cc @JonDouglas

cosmic-blunder commented 1 year ago

imed out because no data was received for 60000ms. Exception of type 'System.TimeoutException' was thrown. Same issue in normal dot net build command on linux.

michha commented 1 year ago

Our organisation also faced this issue on a windows host with windows container.

After experimenting with many dotnet restore options we discovered that a curl on the actual package url showed a data rate of about 2-5 kb/s 🐌. Some curls later we knew the network issue applied to URLs regardless the target network (intranet/www). Probably out of pure luck I discovered https://github.com/microsoft/Windows-Containers/issues/145 . Someone suggested to execute Get-NetAdapterRSC | Disable-NetAdapterRSC on the container host. I gave it a try and didnt even restart the host - just the container. curl was fast 🚀(5 mbit/s) and so was dotnet restore 🥳

TLDR:

slow dotnet restore inside a container could have its root cause outside of nuget/dotnet, try curl a big file inside your container to get a better insight
try Get-NetAdapterRSC | Disable-NetAdapterRSC on your host if you are brave enough

Give 👍 or 👎 on this post if this workaround resolved your issue.

simanga-dev commented 1 year ago

Anyone who know How to fix this on Linux Machine. I am facing the same problem

skotl commented 1 year ago

Don't understand what this does, but I can confirm that it now makes my docker-hosted dotnet restore 100 times faster...!

simanga-dev commented 1 year ago

where do you put this, before or after the build command? @skotl

skotl commented 1 year ago

where do you put this, before or after the build command? @skotl

@h3ndry It's not in the build pipeline, but addressed an issue on my PC where docker was pulling files incredibly slowly. To try this on your PC (and I have genuinely no clue what it does, how safe or dangerous it is, so caveot empor!):

Run PowerShell as an admin
Inside PowerShell execute Get-NetAdapterRSC | Disable-NetAdapterRSC

That's it! The results were instant for me when I reran the docker build on my PC but you may have to reboot your PC ¯_(ツ)_/¯

michha commented 1 year ago

where do you put this, before or after the build command? @skotl

Its a one-time-thing. Do it after/before docker installation

NuGet / Home