search

NuGet / Home

Repo for NuGet Client issues
Other
1.49k stars 253 forks source link

dotnet restore taking up to 12GB in docker container #10251

Closed marcpopMSFT closed 3 years ago

marcpopMSFT commented 3 years ago

Issue moved from dotnet/sdk#14389

From @adam8797 on Friday, October 30, 2020 2:21:38 PM

Hello all,

I've been working to try to determine why my dotnet restore is taking so long in my build system (GitLab CI)

This project is using .net 5 rc2 currently.

I'm restoring three projects, and on developer machines it only ever takes around 30 seconds or so. In the build system it takes about 10 minutes.

The exact restore command is dotnet restore -v n --packages .nuget /clp:PerformanceSummary --disable-parallel

Here is the target performance summary of the latest build:

Target Performance Summary:
        0 ms  ValidateProjects                           1 calls
        0 ms  _GetRestorePackagesPathOverride            1 calls
        0 ms  ValidateToolsVersions                      1 calls
        0 ms  CollectCentralPackageVersions              3 calls
        0 ms  _GenerateRestoreGraphProjectEntry          3 calls
        0 ms  _GetRestoreSettingsCurrentProject          3 calls
        0 ms  CollectFrameworkReferences                 3 calls
        0 ms  _GenerateRestoreProjectPathItemsCurrentProject   3 calls
        0 ms  _GenerateRestoreDependencies               3 calls
        0 ms  _GenerateProjectRestoreGraph               3 calls
        0 ms  _GenerateProjectRestoreGraphCurrentProject   3 calls
        0 ms  CollectPackageReferences                   3 calls
        0 ms  CollectPackageDownloads                    3 calls
        0 ms  _CheckForObsoleteDotNetCliToolReferences   3 calls
        0 ms  _CheckForUnsupportedCppNETCoreVersion      3 calls
        0 ms  _GetProjectJsonPath                        3 calls
        0 ms  _CheckForUnsupportedNETCoreVersion         3 calls
        1 ms  _GetRestoreSettingsPerFramework            3 calls
        1 ms  _CheckForLanguageAndFeatureCombinationSupport   3 calls
        2 ms  _GenerateRestoreSpecs                      3 calls
        2 ms  _GenerateRestoreProjectPathItems           3 calls
        2 ms  _GenerateRestoreProjectSpec                3 calls
        3 ms  _GenerateDotnetCliToolReferenceSpecs       3 calls
        5 ms  _GetRestoreTargetFrameworksOutput          3 calls
        7 ms  _IsProjectRestoreSupported                 3 calls
        9 ms  _LoadRestoreGraphEntryPoints               1 calls
        9 ms  _GenerateProjectRestoreGraphPerFramework   3 calls
       10 ms  ValidateSolutionConfiguration              1 calls
       11 ms  CheckForImplicitPackageReferenceOverrides   3 calls
       13 ms  UpdateAspNetToFrameworkReference           3 calls
       15 ms  ApplyImplicitVersions                      3 calls
       16 ms  _GetRestoreProjectStyle                    3 calls
       26 ms  _GenerateRestoreProjectPathWalk            3 calls
       31 ms  _GenerateRestoreProjectPathItemsPerFramework   3 calls
       32 ms  _GetRestoreSettingsOverrides               3 calls
       45 ms  _GetAllRestoreProjectPathItems             1 calls
       65 ms  _GetRestoreSettings                        3 calls
      162 ms  _GenerateRestoreGraph                      1 calls
      256 ms  ProcessFrameworkReferences                 3 calls
      720 ms  _FilterRestoreGraphProjectInputItems       1 calls
    556229 ms  Restore                                    1 calls
Task Performance Summary:
        1 ms  NuGetMessageTask                           1 calls
        1 ms  WarnForInvalidProjectsTask                 1 calls
        1 ms  GetRestorePackageDownloadsTask             3 calls
        1 ms  GetRestoreFrameworkReferencesTask          3 calls
        2 ms  GetRestoreProjectReferencesTask            3 calls
        2 ms  GetRestoreDotnetCliToolsTask               3 calls
        2 ms  GetRestorePackageReferencesTask            3 calls
        2 ms  RemoveDuplicates                           9 calls
        3 ms  Message                                    8 calls
        4 ms  GetProjectTargetFrameworksTask             3 calls
        5 ms  GetRestoreSolutionProjectsTask             1 calls
        6 ms  ConvertToAbsolutePath                      6 calls
        6 ms  CheckIfPackageReferenceShouldBeFrameworkReference   6 calls
        7 ms  CheckForDuplicateFrameworkReferences       3 calls
        8 ms  GetRestoreProjectStyleTask                 3 calls
        8 ms  ApplyImplicitVersions                      3 calls
        9 ms  CheckForImplicitPackageReferenceOverrides   3 calls
       40 ms  ResolveAppHosts                            3 calls
       64 ms  GetRestoreSettingsTask                     3 calls
      207 ms  ProcessFrameworkReferences                 3 calls
      950 ms  MSBuild                                   11 calls
    556228 ms  RestoreTask                                1 calls

Even weirder, is if I manage to get into the build node and run top I see that dotnet is using 12GB of memory! Output from top:

top - 14:20:54 up 9 days, 22:07,  0 users,  load average: 3.13, 3.58, 3.50
Tasks:  11 total,   1 running,  10 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.1 us,  0.4 sy,  0.0 ni, 94.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 58.5/128309.8 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||                                         ]
MiB Swap:  0.0/0.0      [                                                                                                    ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                   
   33 root      20   0   11.9g   0.2g   0.1g S   2.7   0.2   0:18.10 dotnet
   69 root      20   0    8.2g   0.1g   0.0g S   0.0   0.1   0:00.75 dotnet
   55 root      20   0    8.2g   0.1g   0.0g S   0.0   0.1   0:00.74 dotnet
   14 root      20   0    8.1g   0.1g   0.0g S   0.0   0.1   0:00.61 dotnet
  124 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 bash
  315 root      20   0    0.0g   0.0g   0.0g R   0.0   0.0   0:00.07 top
    6 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 bash
    1 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.02 bash
   12 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 bash
  117 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 sh
  123 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 sh

This seems like an insane amount of memory to be using? How can I reduce this? What more can I provide to help troubleshoot this?

marcpopMSFT commented 3 years ago

Issue moved from dotnet/sdk#14389

From @marcpopMSFT on Friday, November 6, 2020 10:03:11 PM

@nkolev92 @zkat any ideas on this one? Should I move it to the NuGet repo?

marcpopMSFT commented 3 years ago

Issue moved from dotnet/sdk#14389

From @nkolev92 on Saturday, November 7, 2020 2:12:58 AM

Yep, let's move to NuGet.

@adam8797 It'd be helpful if you can create a repro sln. What kind of machines does the build system have? Any special setups we should be aware of?

Things that could affect the memory usage are the number of projects, packages, sources, types of sources etc.

marcpopMSFT commented 3 years ago

Issue moved from dotnet/sdk#14389

From @adam8797 on Saturday, November 7, 2020 2:43:18 AM

@nkolev92 I'm not entirely sure, the images are running in our Kubernetes cluster. I can try to find details if you need specifics. We're using the .net 5 sdk docker image to do the building. We're behind a corporate proxy, but we configure the container with our root certs and proxies in the same way we do for the core 3.1 sdk, which works fine. the build containers have a memory limit of 2GB and 1 CPU to use.

As far as number of projects, Its a plain asp net core app, two projects, maybe 4 or 5 non-microsoft nuget packages

nkolev92 commented 3 years ago

@adam8797 Any change you can provide with a repro? We want to understand whether the scaling is intentional or something that we can consider addressing.

adam8797 commented 3 years ago

I'll see what I can do, but the project was honestly nothing special. Little more than File/New Project. I'll also try with the latest image now that .net 5 is fully out. Will update tomorrow.

nkolev92 commented 3 years ago

Thank you for working with us!

adam8797 commented 3 years ago

I didn't get to do a File/New Project test today, but I did perform a lot of tests with the project I had. I've outlined more about the project structure (its very simple) and the results from today's tests. I think the memory thing may have been a red herring, as I observed the project building in .net core 3.1 take up almost 10 GB of virtual memory, but it still did the restore in a fraction of the time.

I can spend some time tomorrow testing an empty project, I just didn't have the cycles today.

I'm also at work, and I cannot disclose any code we actually use, but I can mock things up for this issue, keeping it as true to reality as possible (changing project names, and such)

Project Structure and Dependencies

Project structure is as follows:

Site.sln
|-- Site/Site.csproj  # ASP.Net Core application
|-- Site.Tests/Site.Tests.csproj  # xUnit based test suite
|-- Site.Closer/Site.Closer.csproj  # Console App
|-- Site.Loader/Site.Loader.csproj  # Console App

External Dependencies of Site:

<ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="5.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="5.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer" Version="4.2.0" />
    <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="5.0.0" />
    <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.10.9" />
    <PackageReference Include="Serilog.Extensions.Logging" Version="3.0.1" />
    <PackageReference Include="Serilog.AspNetCore" Version="3.4.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
    <PackageReference Include="Serilog.Enrichers.CorrelationId" Version="3.0.1" />
    <PackageReference Include="Serilog.Enrichers.Environment" Version="2.1.3" />
    <PackageReference Include="Serilog.Enrichers.Process" Version="2.0.1" />
    <PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
    <PackageReference Include="SomeInternalNuGetPackage" Version="1.0.0" />
    <PackageReference Include="AnotherInternalNugetPackage" Version="1.0.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="5.6.3" />
    <PackageReference Include="Swashbuckle.AspNetCore.Annotations" Version="5.6.3" />
    <PackageReference Include="Swashbuckle.AspNetCore.Newtonsoft" Version="5.6.3" />
</ItemGroup>

External Dependencies of Test Suite

<ItemGroup>
    <PackageReference Include="JunitXml.TestLogger" Version="2.1.78" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.8.0" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
        <PrivateAssets>all</PrivateAssets>
        <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="coverlet.collector" Version="1.3.0">
        <PrivateAssets>all</PrivateAssets>
        <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
</ItemGroup>

The other two projects only depend on Site

Testing

I ran three tests. Keep in mind all of these are running in GitLab CI, in Kubernetes Pods

First, I had previously downgraded the app to core3.1 and I simply had the build system run the following command:

dotnet restore -v m --packages .nuget

Test A: .Net Core 3.1 SDK / ASP.Net Core 3.1 (pulled from dockerhub today)

This is how we've been building for the last few days, since I downgraded to .Net Core 3.1. This is about the performance I expect.

Restored /builds/src/Site.Tests/Site.Tests.csproj (in 7.29 sec).
Restored /builds/src/Site/Site.csproj (in 12.44 sec).
Restored /builds/src/Site.Closer/Site.Closer.csproj (in 5.13 sec).
Restored /builds/src/Site.Loader/Site.Loader.csproj (in 448 ms).

Test B: .Net 5 SDK / ASP.Net Core 3.1 (pulled from dockerhub today)

I actually ran this test last, but it makes more sense as a second test. Upgrading the docker image used to restore, but not touching the project files at all. Interestingly it also seemed to have picked a new restore order.

Restored /builds/src/Site.Loader/Site.Loader.csproj (in 1.78 min).
Restored /builds/src/Site.Closer/Site.Closer.csproj (in 1.78 min).
Restored /builds/src/Site/Site.csproj (in 94 ms).
Restored /builds/src/Site.Tests/Site.Tests.csproj (in 1.61 min).

Test C: .Net 5 SDK / ASP.Net Core 5 (pulled from dockerhub today)

In this test, I upgraded each project to .Net 5. I changed the target framework in each project to net5 then updated all available nuget packages.

Restored /builds/src/Site.Tests/Site.Tests.csproj (in 1.65 min).
Restored /builds/src/Site/Site.csproj (in 1.78 min).
Restored /builds/src/Site.Closer/Site.Closer.csproj (in 7.16 sec).
Restored /builds/src/Site.Loader/Site.Loader.csproj (in 3.95 sec).

Test D: .Net 5 SDK / ASP.Net Core 5 (same container, personal machine, Docker Desktop)

Ran on my local machine, to remove Kubernetes as a factor. Still horribly slow.

Restored /src/Site/src/Site.Closer/Site.Closer.csproj (in 1.85 min).
Restored /src/Site/src/Site.Loader/Site.Loader.csproj (in 1.85 min).
Restored /src/Site/src/Site/Site.csproj (in 132 ms).
Restored /src/Site/src/Site.Tests/Site.Tests.csproj (in 1.61 min).

Test E: .Net 3.1 SDK / ASP.Net Core 3.1 (same container, personal machine, Docker Desktop)

Ran on my local machine, to remove Kubernetes as a factor. Worked as expected.

Restored /src/Site/src/Site.Tests/Site.Tests.csproj (in 15.54 sec).
Restored /src/Site/src/Site/Site.csproj (in 17.51 sec).
Restored /src/Site/src/Site.Closer/Site.Closer.csproj (in 1.92 sec).
Restored /src/Site/src/Site.Loader/Site.Loader.csproj (in 666 ms).

Build Containers

The containers we build in are slightly specialized. We base our SDK containers off your public sdk images, and we have a Dockerfile that adds our own dependencies (certs, proxies, etc.) This script is exactly the same for Core31 and Net5 sdk images.

As you can see, the baseimage is passed in as an argument, meaning that our 3.1sdk and our 5sdk images are identical, except for the base image we get from dockerhub.

See the redacted script here:

ARG BASEIMAGE
FROM $BASEIMAGE
COPY *.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates \
    && export http_proxy=http://http-proxy:80 \
    && export https_proxy=$http_proxy \
    && export HTTP_PROXY=$http_proxy \
    && export HTTPS_PROXY=$http_proxy \
    && export NODE_TLS_REJECT_UNAUTHORIZED=0 \
    && apt-get update \
    && apt-get install -y zip unzip gss-ntlmssp htop procps \
    && echo ------ Setup NuGet ------ \
    && dotnet nuget add source https://our.local.artifactory.repo.with.nuget.org.forwarding/api/nuget/virtual -n Virtual -u user -p password --store-password-in-clear-text \
    && dotnet nuget disable source nuget.org \ 
    && echo ------ Install and configure NPM ------ \
    && curl -sL https://deb.nodesource.com/setup_14.x | bash - \
    && apt-get install -y nodejs \
    && npm config set strict-ssl false \
    && npm config set proxy $http_proxy \
    && npm config set https-proxy $http_proxy \
    && echo ------ Install and configure Yarn ------ \
    && npm install -g yarn \
    && yarn config set strict-ssl false \
    && yarn config set proxy $http_proxy \
    && yarn config set https-proxy $http_proxy
adam8797 commented 3 years ago

@nkolev92 Hey there, just wanted to bump this. Do you need me to provide any more information?

nkolev92 commented 3 years ago

Hey @adam8797,

From what I can see from the set-up, when the SDK changes there are some significant changes in what gets download.

Each SDK ships with the targeting packs it's own runtime version, but for any other version, it will spend some time downloading extra packages.

The runtime/targeting packs are pretty large and they can take some time downloading.

I imagine in all these cases you are dealing with an empty machine, is that correct?

You can consider looking at the global packages folder size and file count.

Beyond that, as you mentioned earlier, seems like the memory consumption is in line with what build would do and something intentional.

adam8797 commented 3 years ago

@nkolev92 Yup, we start a new empty container each time. And the targeting thing makes sense, but that doesn't explain the slowness when the SDK matches the target. See my Tests C and D, where I'm building .net5 with the .net5 sdk.

nkolev92 commented 3 years ago

@adam8797

Capturing the number of packages installed/download is probably valuable.

adam8797 commented 3 years ago

@nkolev92 any easy way to do this, or should I just grab the output at a more verbose setting?

nkolev92 commented 3 years ago

Verbose logs would do it.

Alternatively, you can glob for *.nupkg files in the global packages folder.

adam8797 commented 3 years ago

Hey there @nkolev92. Sorry, my bad for letting this one go cold. We ended up going back to core3.1 for a while.

I did just now try to migrate again to net5 and I'm having very similar results.

Package restores that took ~15-20 sec are now taking ~2-3 mins

Here is a glob of the .nuget folder at project root of the net5 version of the project

root@runner-7m3xvgyy-project-1082-concurrent-0jfdns:/builds/root/src/.nuget# du -sh *
84K     app.metrics
424K    app.metrics.abstractions
100K    app.metrics.aspnetcore
84K     app.metrics.aspnetcore.abstractions
80K     app.metrics.aspnetcore.all
144K    app.metrics.aspnetcore.core
156K    app.metrics.aspnetcore.endpoints
96K     app.metrics.aspnetcore.hosting
84K     app.metrics.aspnetcore.mvc
104K    app.metrics.aspnetcore.mvc.core
88K     app.metrics.aspnetcore.routing
168K    app.metrics.aspnetcore.tracking
136K    app.metrics.concurrency
704K    app.metrics.core
68K     app.metrics.extensions.configuration
72K     app.metrics.extensions.dependencyinjection
68K     app.metrics.extensions.healthchecks
80K     app.metrics.extensions.hosting
196K    app.metrics.formatters.ascii
144K    app.metrics.formatters.influxdb
192K    app.metrics.formatters.json
144K    app.metrics.reporting.influxdb
856K    automapper
148K    automapper.collection
64K     automapper.extensions.microsoft.dependencyinjection
2.7M    castle.core
884K    colorful.console
6.4M    coverlet.collector
2.5M    csvhelper
2.0M    fakeiteasy
192K    flurl
336K    jsonsubtypes
148K    junitxml.testlogger
92K     microsoft.aspnetcore.hosting.abstractions
80K     microsoft.aspnetcore.hosting.server.abstractions
180K    microsoft.aspnetcore.http
176K    microsoft.aspnetcore.http.abstractions
108K    microsoft.aspnetcore.http.features
256K    microsoft.aspnetcore.jsonpatch
184K    microsoft.aspnetcore.mvc.newtonsoftjson
200K    microsoft.aspnetcore.mvc.razor.extensions
196K    microsoft.aspnetcore.mvc.razor.runtimecompilation
728K    microsoft.aspnetcore.mvc.versioning
300K    microsoft.aspnetcore.mvc.versioning.apiexplorer
1.4M    microsoft.aspnetcore.razor.language
156K    microsoft.aspnetcore.webutilities
504K    microsoft.bcl.asyncinterfaces
3.6M    microsoft.codeanalysis.analyzers
12M     microsoft.codeanalysis.common
34M     microsoft.codeanalysis.csharp
204K    microsoft.codeanalysis.razor
18M     microsoft.codecoverage
8.6M    microsoft.csharp
11M     microsoft.data.edm
35M     microsoft.data.odata
9.7M    microsoft.data.services.client
120K    microsoft.dotnet.platformabstractions
1.4M    microsoft.extensions.apidescription.server
564K    microsoft.extensions.configuration
896K    microsoft.extensions.configuration.abstractions
760K    microsoft.extensions.configuration.binder
2.0M    microsoft.extensions.dependencyinjection
1.3M    microsoft.extensions.dependencyinjection.abstractions
924K    microsoft.extensions.dependencymodel
88K     microsoft.extensions.diagnostics.healthchecks.abstractions
356K    microsoft.extensions.fileproviders.abstractions
412K    microsoft.extensions.hosting.abstractions
568K    microsoft.extensions.http
224K    microsoft.extensions.http.polly
780K    microsoft.extensions.logging
1.1M    microsoft.extensions.logging.abstractions
84K     microsoft.extensions.objectpool
1.9M    microsoft.extensions.options
400K    microsoft.extensions.options.configurationextensions
1.6M    microsoft.extensions.primitives
168K    microsoft.net.http.headers
156K    microsoft.net.test.sdk
516K    microsoft.netcore.platforms
152K    microsoft.netcore.targets
776K    microsoft.openapi
6.7M    microsoft.testplatform.objectmodel
22M     microsoft.testplatform.testhost
1.2M    microsoft.visualbasic
2.6M    microsoft.visualstudio.azure.containers.tools.targets
352K    microsoft.win32.primitives
1.6M    microsoft.win32.registry
2.2M    mstest.testframework
584K    neo4j.driver.signed
460K    neo4jclient
96K     neo4jclient.returnpoly
68K     netstandard.library
28M     newtonsoft.json
608K    newtonsoft.json.bson
1.8M    newtonsoft.json.schema
568K    nuget.frameworks
2.8M    polly
60K     polly.extensions.http
336K    runtime.debian.8-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.fedora.23-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.fedora.24-x64.runtime.native.system.security.cryptography.openssl
120K    runtime.native.system
92K     runtime.native.system.data.sqlclient.sni
120K    runtime.native.system.io.compression
120K    runtime.native.system.net.http
120K    runtime.native.system.net.security
60K     runtime.native.system.security.cryptography
60K     runtime.native.system.security.cryptography.apple
128K    runtime.native.system.security.cryptography.openssl
336K    runtime.opensuse.13.2-x64.runtime.native.system.security.cryptography.openssl
336K    runtime.opensuse.42.1-x64.runtime.native.system.security.cryptography.openssl
80K     runtime.osx.10.10-x64.runtime.native.system.security.cryptography.apple
296K    runtime.osx.10.10-x64.runtime.native.system.security.cryptography.openssl
336K    runtime.rhel.7-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.ubuntu.14.04-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.ubuntu.16.04-x64.runtime.native.system.security.cryptography.openssl
336K    runtime.ubuntu.16.10-x64.runtime.native.system.security.cryptography.openssl
316K    runtime.win-arm64.runtime.native.system.data.sqlclient.sni
308K    runtime.win-x64.runtime.native.system.data.sqlclient.sni
276K    runtime.win-x86.runtime.native.system.data.sqlclient.sni
5.5M    serilog
128K    serilog.aspnetcore
244K    serilog.enrichers.correlationid
68K     serilog.enrichers.environment
56K     serilog.enrichers.process
72K     serilog.enrichers.thread
376K    serilog.expressions
144K    serilog.extensions.hosting
64K     serilog.extensions.logging
152K    serilog.formatting.compact
244K    serilog.settings.configuration
180K    serilog.sinks.console
136K    serilog.sinks.debug
208K    serilog.sinks.file
96K     serilog.sinks.periodicbatching
80K     serilog.sinks.redis.core
128K    serilog.sinks.redis.list
1.7M    redacted.library
28K     redacted.library
28K     redacted.library
200K    redacted.library
76K     redacted.library
96K     redacted.library
40K     redacted.library
28K     redacted.library
244K    redacted.library
60K     redacted.library
44K     redacted.library
48K     redacted.library
32K     redacted.library
644K    redacted.library
2.1M    stackexchange.redis
388K    superpower
40K     swashbuckle.aspnetcore
240K    swashbuckle.aspnetcore.annotations
200K    swashbuckle.aspnetcore.newtonsoft
196K    swashbuckle.aspnetcore.swagger
580K    swashbuckle.aspnetcore.swaggergen
14M     swashbuckle.aspnetcore.swaggerui
784K    system.appcontext
408K    system.buffers
1.9M    system.collections
1.6M    system.collections.concurrent
1.7M    system.collections.immutable
920K    system.collections.nongeneric
352K    system.collections.specialized
232K    system.componentmodel
2.7M    system.componentmodel.annotations
368K    system.componentmodel.eventbasedasync
260K    system.componentmodel.primitives
704K    system.componentmodel.typeconverter
488K    system.console
16M     system.data.sqlclient
520K    system.diagnostics.debug
652K    system.diagnostics.diagnosticsource
1.2M    system.diagnostics.process
232K    system.diagnostics.textwritertracelistener
312K    system.diagnostics.tools
492K    system.diagnostics.tracesource
1.7M    system.diagnostics.tracing
2.1M    system.dynamic.runtime
1.5M    system.globalization
592K    system.globalization.calendars
768K    system.globalization.extensions
1.8M    system.io
2.1M    system.io.compression
264K    system.io.compression.zipfile
776K    system.io.filesystem
464K    system.io.filesystem.primitives
1.9M    system.linq
14M     system.linq.async
6.6M    system.linq.expressions
520K    system.linq.queryable
1.3M    system.memory
5.6M    system.net.http
1.3M    system.net.http.winhttphandler
1.1M    system.net.nameresolution
1.8M    system.net.primitives
648K    system.net.requests
1.8M    system.net.security
624K    system.net.sockets
180K    system.net.webheadercollection
1.2M    system.objectmodel
15M     system.private.servicemodel
12M     system.reactive
1.9M    system.reflection
356K    system.reflection.dispatchproxy
1.7M    system.reflection.emit
480K    system.reflection.emit.ilgeneration
392K    system.reflection.emit.lightweight
376K    system.reflection.extensions
5.9M    system.reflection.metadata
760K    system.reflection.primitives
1.0M    system.reflection.typeextensions
344K    system.resources.resourcemanager
16M     system.runtime
816K    system.runtime.compilerservices.unsafe
2.6M    system.runtime.extensions
240K    system.runtime.handles
5.9M    system.runtime.interopservices
1.1M    system.runtime.interopservices.runtimeinformation
1.1M    system.runtime.numerics
232K    system.runtime.serialization.formatters
984K    system.runtime.serialization.primitives
1.7M    system.security.accesscontrol
680K    system.security.claims
2.4M    system.security.cryptography.algorithms
1.8M    system.security.cryptography.cng
712K    system.security.cryptography.csp
768K    system.security.cryptography.encoding
436K    system.security.cryptography.openssl
488K    system.security.cryptography.primitives
2.9M    system.security.cryptography.x509certificates
464K    system.security.principal
3.4M    system.security.principal.windows
376K    system.servicemodel.duplex
560K    system.servicemodel.http
500K    system.servicemodel.nettcp
1.1M    system.servicemodel.primitives
348K    system.servicemodel.security
3.8M    system.spatial
880K    system.text.encoding
10M     system.text.encoding.codepages
720K    system.text.encoding.extensions
276K    system.text.encodings.web
1.3M    system.text.json
1.9M    system.text.regularexpressions
1.9M    system.threading
2.0M    system.threading.tasks
596K    system.threading.tasks.extensions
472K    system.threading.thread
448K    system.threading.threadpool
328K    system.threading.timer
1.2M    system.valuetuple
5.4M    system.xml.readerwriter
1.9M    system.xml.xdocument
540K    system.xml.xmldocument
56K     xunit
180K    xunit.abstractions
360K    xunit.analyzers
172K    xunit.assert
72K     xunit.core
656K    xunit.extensibility.core
844K    xunit.extensibility.execution
2.4M    xunit.runner.visualstudio
1.9M    yamldotnet

I'm currently battling some build issues with the 3.1 version, but I can probably glob that as well if you think it might be helpful

nkolev92 commented 3 years ago

The thing that I'm trying to figure out is if the discrepancy is because of the sheer number of packages that are getting downloaded in the net5 graph.

So isolating the global packages folder and globbing for files would help us understand that.

adam8797 commented 3 years ago

Fair enough. I’ll see if I can get you a glob of the 3.1 version during the work day.

On Mar 22, 2021, at 5:29 PM, Nikolche Kolev @.***> wrote:

 The thing that I'm trying to figure out is if the discrepancy is because of the sheer number of packages that are getting downloaded in the net5 graph.

So isolating the global packages folder and globbing for files would help us understand that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

adam8797 commented 3 years ago

Ok, I have a glob from core 3.1. Its from an older commit, but should have largely the same packages in it

root@runner-7m3xvgyy-project-1082-concurrent-0ztzsp:/builds/root/src/.nuget# du -sh *
2.7M    castle.core
2.6M    coverlet.collector
1.1M    csvhelper
2.4M    fakeiteasy
192K    flurl
336K    jsonsubtypes
148K    junitxml.testlogger
180K    microsoft.aspnetcore.http
176K    microsoft.aspnetcore.http.abstractions
108K    microsoft.aspnetcore.http.features
160K    microsoft.aspnetcore.jsonpatch
180K    microsoft.aspnetcore.mvc.newtonsoftjson
184K    microsoft.aspnetcore.mvc.razor.extensions
192K    microsoft.aspnetcore.mvc.razor.runtimecompilation
348K    microsoft.aspnetcore.mvc.versioning
156K    microsoft.aspnetcore.mvc.versioning.apiexplorer
1.4M    microsoft.aspnetcore.razor.language
156K    microsoft.aspnetcore.webutilities
280K    microsoft.bcl.asyncinterfaces
1.2M    microsoft.codeanalysis.analyzers
5.4M    microsoft.codeanalysis.common
16M     microsoft.codeanalysis.csharp
188K    microsoft.codeanalysis.razor
9.2M    microsoft.codecoverage
6.6M    microsoft.csharp
11M     microsoft.data.edm
35M     microsoft.data.odata
9.7M    microsoft.data.services.client
120K    microsoft.dotnet.platformabstractions
1.4M    microsoft.extensions.apidescription.server
400K    microsoft.extensions.configuration
552K    microsoft.extensions.configuration.abstractions
380K    microsoft.extensions.configuration.binder
1.5M    microsoft.extensions.dependencyinjection
848K    microsoft.extensions.dependencyinjection.abstractions
792K    microsoft.extensions.dependencymodel
92K     microsoft.extensions.fileproviders.abstractions
92K     microsoft.extensions.hosting.abstractions
248K    microsoft.extensions.http
112K    microsoft.extensions.http.polly
448K    microsoft.extensions.logging
668K    microsoft.extensions.logging.abstractions
84K     microsoft.extensions.objectpool
1012K   microsoft.extensions.options
68K     microsoft.extensions.options.configurationextensions
1.1M    microsoft.extensions.primitives
168K    microsoft.net.http.headers
96K     microsoft.net.test.sdk
516K    microsoft.netcore.platforms
152K    microsoft.netcore.targets
776K    microsoft.openapi
2.9M    microsoft.testplatform.objectmodel
15M     microsoft.testplatform.testhost
1.2M    microsoft.visualbasic
2.6M    microsoft.visualstudio.azure.containers.tools.targets
352K    microsoft.win32.primitives
1.6M    microsoft.win32.registry
2.2M    mstest.testframework
580K    neo4j.driver.signed
468K    neo4jclient
96K     neo4jclient.returnpoly
68K     netstandard.library
28M     newtonsoft.json
608K    newtonsoft.json.bson
568K    nuget.frameworks
2.0M    polly
60K     polly.extensions.http
336K    runtime.debian.8-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.fedora.23-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.fedora.24-x64.runtime.native.system.security.cryptography.openssl
120K    runtime.native.system
92K     runtime.native.system.data.sqlclient.sni
120K    runtime.native.system.io.compression
120K    runtime.native.system.net.http
120K    runtime.native.system.net.security
60K     runtime.native.system.security.cryptography
60K     runtime.native.system.security.cryptography.apple
128K    runtime.native.system.security.cryptography.openssl
336K    runtime.opensuse.13.2-x64.runtime.native.system.security.cryptography.openssl
336K    runtime.opensuse.42.1-x64.runtime.native.system.security.cryptography.openssl
80K     runtime.osx.10.10-x64.runtime.native.system.security.cryptography.apple
296K    runtime.osx.10.10-x64.runtime.native.system.security.cryptography.openssl
336K    runtime.rhel.7-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.ubuntu.14.04-x64.runtime.native.system.security.cryptography.openssl
328K    runtime.ubuntu.16.04-x64.runtime.native.system.security.cryptography.openssl
336K    runtime.ubuntu.16.10-x64.runtime.native.system.security.cryptography.openssl
316K    runtime.win-arm64.runtime.native.system.data.sqlclient.sni
308K    runtime.win-x64.runtime.native.system.data.sqlclient.sni
276K    runtime.win-x86.runtime.native.system.data.sqlclient.sni
5.5M    serilog
132K    serilog.aspnetcore
244K    serilog.enrichers.correlationid
68K     serilog.enrichers.environment
56K     serilog.enrichers.process
72K     serilog.enrichers.thread
220K    serilog.expressions
56K     serilog.extensions.hosting
64K     serilog.extensions.logging
152K    serilog.formatting.compact
128K    serilog.formatting.elasticsearch
244K    serilog.settings.configuration
180K    serilog.sinks.console
84K     serilog.sinks.debug
208K    serilog.sinks.file
96K     serilog.sinks.periodicbatching
80K     serilog.sinks.redis.core
128K    serilog.sinks.redis.list
828K    redacted.library
28K     redacted.library
28K     redacted.library
76K     redacted.library
96K     redacted.library
40K     redacted.library
28K     redacted.library
244K    redacted.library
32K     redacted.library
44K     redacted.library
644K    redacted.library
2.1M    stackexchange.redis
388K    superpower
40K     swashbuckle.aspnetcore
168K    swashbuckle.aspnetcore.annotations
144K    swashbuckle.aspnetcore.newtonsoft
140K    swashbuckle.aspnetcore.swagger
392K    swashbuckle.aspnetcore.swaggergen
8.3M    swashbuckle.aspnetcore.swaggerui
784K    system.appcontext
408K    system.buffers
1.9M    system.collections
1.6M    system.collections.concurrent
2.6M    system.collections.immutable
920K    system.collections.nongeneric
352K    system.collections.specialized
232K    system.componentmodel
2.6M    system.componentmodel.annotations
368K    system.componentmodel.eventbasedasync
260K    system.componentmodel.primitives
704K    system.componentmodel.typeconverter
244K    system.console
16M     system.data.sqlclient
520K    system.diagnostics.debug
652K    system.diagnostics.diagnosticsource
1.2M    system.diagnostics.process
312K    system.diagnostics.tools
492K    system.diagnostics.tracesource
1.7M    system.diagnostics.tracing
2.1M    system.dynamic.runtime
1.5M    system.globalization
592K    system.globalization.calendars
768K    system.globalization.extensions
1.8M    system.io
2.1M    system.io.compression
264K    system.io.compression.zipfile
776K    system.io.filesystem
464K    system.io.filesystem.primitives
1.9M    system.linq
6.7M    system.linq.async
6.6M    system.linq.expressions
520K    system.linq.queryable
1.8M    system.memory
5.6M    system.net.http
1.3M    system.net.http.winhttphandler
1.1M    system.net.nameresolution
1.8M    system.net.primitives
648K    system.net.requests
1.8M    system.net.security
624K    system.net.sockets
180K    system.net.webheadercollection
1.2M    system.objectmodel
15M     system.private.servicemodel
12M     system.reactive
1.9M    system.reflection
356K    system.reflection.dispatchproxy
680K    system.reflection.emit
480K    system.reflection.emit.ilgeneration
392K    system.reflection.emit.lightweight
376K    system.reflection.extensions
3.0M    system.reflection.metadata
760K    system.reflection.primitives
1.0M    system.reflection.typeextensions
344K    system.resources.resourcemanager
16M     system.runtime
1.1M    system.runtime.compilerservices.unsafe
2.6M    system.runtime.extensions
240K    system.runtime.handles
5.9M    system.runtime.interopservices
1.1M    system.runtime.interopservices.runtimeinformation
1.1M    system.runtime.numerics
232K    system.runtime.serialization.formatters
984K    system.runtime.serialization.primitives
1.7M    system.security.accesscontrol
680K    system.security.claims
2.4M    system.security.cryptography.algorithms
1.8M    system.security.cryptography.cng
712K    system.security.cryptography.csp
768K    system.security.cryptography.encoding
436K    system.security.cryptography.openssl
488K    system.security.cryptography.primitives
2.9M    system.security.cryptography.x509certificates
464K    system.security.principal
3.4M    system.security.principal.windows
376K    system.servicemodel.duplex
560K    system.servicemodel.http
500K    system.servicemodel.nettcp
1.1M    system.servicemodel.primitives
348K    system.servicemodel.security
3.8M    system.spatial
880K    system.text.encoding
10M     system.text.encoding.codepages
720K    system.text.encoding.extensions
276K    system.text.encodings.web
1.3M    system.text.json
1.9M    system.text.regularexpressions
1.9M    system.threading
2.0M    system.threading.tasks
592K    system.threading.tasks.extensions
472K    system.threading.thread
448K    system.threading.threadpool
328K    system.threading.timer
1.2M    system.valuetuple
5.4M    system.xml.readerwriter
1.9M    system.xml.xdocument
540K    system.xml.xmldocument
56K     xunit
180K    xunit.abstractions
360K    xunit.analyzers
172K    xunit.assert
72K     xunit.core
656K    xunit.extensibility.core
844K    xunit.extensibility.execution
2.4M    xunit.runner.visualstudio
1.9M    yamldotnet
adam8797 commented 3 years ago

Doing a diff between them shows a few additions, but nothing huge being added. And I can tell you that most of whats new was added after this problem had already been discovered

$ diff 31.txt 5.txt
0a1,25
> app.metrics
> app.metrics.abstractions
> app.metrics.aspnetcore
> app.metrics.aspnetcore.abstractions
> app.metrics.aspnetcore.all
> app.metrics.aspnetcore.core
> app.metrics.aspnetcore.endpoints
> app.metrics.aspnetcore.hosting
> app.metrics.aspnetcore.mvc
> app.metrics.aspnetcore.mvc.core
> app.metrics.aspnetcore.routing
> app.metrics.aspnetcore.tracking
> app.metrics.concurrency
> app.metrics.core
> app.metrics.extensions.configuration
> app.metrics.extensions.dependencyinjection
> app.metrics.extensions.healthchecks
> app.metrics.extensions.hosting
> app.metrics.formatters.ascii
> app.metrics.formatters.influxdb
> app.metrics.formatters.json
> app.metrics.reporting.influxdb
> automapper
> automapper.collection
> automapper.extensions.microsoft.dependencyinjection
1a27
> colorful.console
7a34,35
> microsoft.aspnetcore.hosting.abstractions
> microsoft.aspnetcore.hosting.server.abstractions
36a65
> microsoft.extensions.diagnostics.healthchecks.abstractions
64a94
> newtonsoft.json.schema
100d129
< serilog.formatting.elasticsearch
118a148,150
> redacted.library
> redacted.library
> redacted.library
143a176
> system.diagnostics.textwritertracelistener
adam8797 commented 3 years ago

Hey there @nkolev92 I think we can take a step towards closing this one.

After some work with Fiddler I discovered that while building the container was issuing dozens and dozens of requests to http://ts-crl.ws.symantec.com/sha256-tss-ca.crl. Our build hosts are isolated from the internet and need an http_proxy and https_proxy set in order to communicate.

Setting these variables directly before the build in the Dockerfile e.g RUN export http_proxy=proxy:80; export https_proxy=$http_proxy; dotnet build ... allowed the requests to fail-fast and our build time is back down to what is expected.

My theory is that the .net core 3.1 sdk isn't checking this, or fails once then skips.

I'm hoping that setting the proxy then allowing this URL through the firewall may help bring the build times down even more.

nkolev92 commented 3 years ago

My theory is that the .net core 3.1 sdk isn't checking this, or fails once then skips.

I think you are right. The 5.0 SDK does signature verification, while the 3.1 did not.

fyi @heng-liu @kartheekp-ms

Thanks for following up.

adam8797 commented 3 years ago

Just a follow up for posterity:

Adding the exception to the corp firewall for that symantec url cut the time spent restoring packages again by 50%. Same project that used to take ~5-10 minutes to restore now takes 10 seconds.