dotnet / msbuild

The Microsoft Build Engine (MSBuild) is the build platform for .NET and Visual Studio.
https://docs.microsoft.com/visualstudio/msbuild/msbuild
MIT License
5.23k stars 1.35k forks source link

BuildSubmission.Execute() starts one MsBuild child process per project to compile (without re-usage) #3420

Open Pilchie opened 6 years ago

Pilchie commented 6 years ago

From @ocox1 on June 18, 2018 15:23

My tool gives our developers the chance to compile all our projects (~1000 projects) on their local machines. So we have a build.proj with these nodes:

In my application I try to compile these projects (contained in the ProjectList) by calling buildSubmission.Execute(). This worked fine with VisualStudio 2015 / MsBuild 14, but after migration to VisualStudio 2017 / MsBuild 15 (15.7.179) for every project from the ProjectList a new MsBuild instance is created in the background (1000 child processes in my case). After a short time I have to kill my application before my computer collapses because more and more resources are used (some processes crashes with OutOfMemory, Explorer.exe freezes...). If I kill my application early enough, the MsBuild processes keep in memory for ~15 minutes.

Here is a code snippet:

ProjectCollection projectCol = new ProjectCollection();

BuildParameters buildParams = new BuildParameters(projectCol)
{
    BuildThreadPriority = ThreadPriority.Highest,
    EnableNodeReuse = true,
    MaxNodeCount = Convert.ToInt32(Environment.GetEnvironmentVariable("NUMBER_OF_PROCESSORS"))
};

BuildManager bm = new BuildManager();
bm.BeginBuild(buildParams);

BuildRequestData buildReq = new BuildRequestData(
                                                               projectPath, // path to build.proj
                                                               new Dictionary<string, string>
                                                                                   { { "Configuration", "Debug" } }, // buildProperties
                                                               "15.0",  // toolsVersion
                                                               new string[] { "_Make" }, // targetsToBuild
                                                               null);
BuildSubmission buildSubmission = bm.PendBuildRequest(buildReq);
buildSubmission.Execute();
//buildSubmission.ExecuteAsync(null, null); // same result with ExecuteAsync

// clean up resources...

msbuildprocesses

Copied from original issue: dotnet/project-system#3665

ocox1 commented 6 years ago

Additional information:

ocox1 commented 6 years ago

Attached are the demo project and my build.prj file. MsBuildConsole.zip MyBuild.zip

rainersigwald commented 6 years ago

Can you let us know if https://docs.microsoft.com/en-us/visualstudio/msbuild/updating-an-existing-application does not help you resolve this?

ocox1 commented 6 years ago

Thanks @rainersigwald for your response. I tried the proposed way. The results are inconsistent - therefore it took a while for my answer:

There are the following 5 failure hotspots for project build failures:

  1. task "MarkupCompilePass1" failes with: C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Microsoft.WinFx.targets(268,9): error MC1000: Unknown build error, 'Object reference not set to an instance of an object.' (The project contains WPF xaml files in this case, but other WPF projects succeeds).

  2. Done executing task "GenerateResource" -- FAILED. Done building target "CoreResGen" in project "XYZ.csproj" -- FAILED.

  3. Target ResolveComReferences: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2725,5): error MSB4061: The "ResolveComReference" task could not be instantiated from "Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a". Could not load file or assembly 'Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2725,5): error MSB4060: The "ResolveComReference" task has been declared or used incorrectly, or failed during construction. Check the spelling of the task name and the assembly name.

  4. Target _CopyFilesMarkedCopyLocal: C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Microsoft.WinFx.targets(268,9): error MC1000: Unknown build error, 'Object reference not set to an instance of an object.'

  5. Target GenerateTargetFrameworkMonikerAttribute: Skipping target "GenerateTargetFrameworkMonikerAttribute" because all output files are up-to-date with respect to the input files. C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Microsoft.WinFx.targets(268,9): error MC1000: Unknown build error, 'Object reference not set to an instance of an object.'

As you see this is a mixture of failure reasons, but failed project may compile successfully in next run. And I'm sure my ~1100 projects are not the root cause of these compiler failures. But probably these 5 failure hotspots point in the right direction to solve the problem.