Out of Memory on Simple project with dotnet restore

[fermoyboy]

I am running dotnet restore as part of a Jenkins build pipeline. The restore often fails with variations on OutOfMemory. I initially increased the 8Gb RAM on the machine to 16Gb, but this had no effect.

DeptXWindowsService.sln is relatively straightforward, containing c. 12 projects. None of these projects are unusual, excepting perhaps that they are all .NET Framework 4.8 based. They all build on Visual Studio 2022 without problems.

To recreate the situation, I ran two command-line processes on the build machine, both of which failed. The command lines were:

> D:\Build_Pipeline>dotnet restore -v diag Client\DeptXWindowsService.sln 1> dotnet-restore.txt 2>&1

> D:\Build_Pipeline>dotnet restore --disable-parallel -v diag Client\DeptXWindowsService.sln 1> dotnet-restore-disable-parallel.txt 2>&1

dotnet-restore.txt dotnet-restore-disable-parallel.txt

This created the two outputs attached. Both processes exited with a failure after apparently running out of memory, displaying message boxes or stderr errors indicating the out of memory exception, albeit slightly different variations. I imagine this just depends on exactly where the process ran out of memory. Running identical commands back-to-back may even fail in slightly different ways, but they all amount to "out of memory".

Memory usage is only around 5% of the 16Gb before running these processes and the machine is perfectly normal in set-up in other regards, such as normal amount of swap space. The main build task runs without requiring or using even a fraction of this memory.

What can I do to stop this happening? The key issue is not "What happens when it runs out of memory", but rather "Why is a simple dotnet restore using so much memory?"

Versions Information:

D:\Build_Pipeline>msbuild /version
MSBuild version 17.4.1+9a89d02ff for .NET Framework
17.4.1.60106
D:\Build_Pipeline>dotnet --version
7.0.102

systeminfo.txt

[ghost]

Issue is missing Type label, remember to add a Type label

[jeffkl]

@rainersigwald who would be the best person to look into this from MSBuild? The logs show that nodes couldn't be launched which seems very odd...

MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
Microsoft.Build.Framework.InternalErrorException: MSB0001: Internal MSBuild Error: Cannot acquire required number of nodes.
=============
System.AggregateException: One or more errors occurred. (Not enough storage is available to process this command.)
 ---> Microsoft.Build.BackEnd.NodeFailedToLaunchException: Not enough storage is available to process this command.
   at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs)
   at Microsoft.Build.BackEnd.NodeLauncher.DisableMSBuildServer(Func`1 func)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>g__StartNewNode|2(Int32 nodeId)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>b__0(Int32 nodeId)
   --- End of inner exception stack trace ---

 ---> System.AggregateException: One or more errors occurred. (Not enough storage is available to process this command.)
 ---> Microsoft.Build.BackEnd.NodeFailedToLaunchException: Not enough storage is available to process this command.
   at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs)
   at Microsoft.Build.BackEnd.NodeLauncher.DisableMSBuildServer(Func`1 func)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>g__StartNewNode|2(Int32 nodeId)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>b__0(Int32 nodeId)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Microsoft.Build.Shared.ErrorUtilities.ThrowInternalError(String message, Exception innerException, Object[] args)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.GetNodes(String msbuildLocation, String commandLineArgs, Int32 nextNodeId, INodePacketFactory factory, Handshake hostHandshake, NodeContextCreatedDelegate createNode, NodeContextTerminateDelegate terminateNode, Int32 numberOfNodesToCreate)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProc.CreateNodes(Int32 nextNodeId, INodePacketFactory factory, Func`2 configurationFactory, Int32 numberOfNodesToCreate)
   at Microsoft.Build.BackEnd.NodeManager.AttemptCreateNode(INodeProvider nodeProvider, NodeConfiguration nodeConfiguration, Int32 numberOfNodesToCreate)
   at Microsoft.Build.BackEnd.NodeManager.CreateNodes(NodeConfiguration configuration, NodeAffinity nodeAffinity, Int32 numberOfNodesToCreate)
   at Microsoft.Build.Execution.BuildManager.PerformSchedulingActions(IEnumerable`1 responses)
   at Microsoft.Build.Execution.BuildManager.ProcessPacket(Int32 node, INodePacket packet)
   at Microsoft.Build.Execution.BuildManager.ProcessWorkQueue(Action action)
--- End of stack trace from previous location ---
   at Microsoft.Build.Execution.BuildManager.EndBuild()
   at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, TextWriter targetsWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsNotAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, GraphBuildOptions graphBuildOptions, Boolean lowPriority, String[] inputResultsCaches, String outputResultsCache, String[] commandLine)

[aortiz-msft]

This issue was moved to dotnet/msbuild#8466