search

dotnet / msbuild

The Microsoft Build Engine (MSBuild) is the build platform for .NET and Visual Studio.
https://docs.microsoft.com/visualstudio/msbuild/msbuild
MIT License
5.22k stars 1.35k forks source link

Serialization of Targets when awaiting for a single target #7864

Open yuehuang010 opened 2 years ago

yuehuang010 commented 2 years ago

Issue Description

An behavior when building a "Fast" and "Slow" target, when a project reference for "Fast" target, the build waits for both "Fast" and "Slow" to finish. This blocks P2P from continuing when "Fast" is done.

image

With a bad workaround, image

Steps to Reproduce

Attached binlog with projects. BuildWorkers.zip

msbuild.exe /m Solution.proj

Expected Behavior

I expect "Slow" targets from both worker projects to run in parallel. The "fast" target will stagger because of the project dependency.

Actual Behavior

I see that Worker1.proj runs "Fast+Slow" targets, then Worker2.proj "Fast+Slow" in serialization.

Analysis

No Idea.

Versions & Configurations

Repros in VS2022 and VS2019, probably even earlier.

AR-May commented 2 years ago

@rainersigwald, could you please take a look?

CDAlexanderW commented 2 years ago

Any idea when this will be investigated? This is important for us!

thx!

rainersigwald commented 2 years ago

Any idea when this will be investigated? This is important for us!

Can you please explain why it's important for you? This is extremely long-standing MSBuild behavior and the fix will not be easy, but it hasn't been at the top of the priority list because it's not generally super impactful. So I'd love to get data otherwise.

CDAlexanderW commented 2 years ago

Any idea when this will be investigated? This is important for us!

Can you please explain why it's important for you? This is extremely long-standing MSBuild behavior and the fix will not be easy, but it hasn't been at the top of the priority list because it's not generally super impactful. So I'd love to get data otherwise.

It looks like this may block AllowParallelCompileInReferencedProjects from working correctly.

yuehuang010 commented 2 years ago

Hi, providing some context. AllowParallelCompileInReferencedProjects is an experimental feature to speed vcxproj when building from the command line. It splits the project into 3 passes that don't have project reference dependency on similar passes. This in turns allows Pass 1 to run in parallel with other projects Pass 1. Pass 2 depend its P2P's Pass 1. (likewise, Pass 3 depends on Pass 2) To make the dependency, Pass 2 invokes MSBuild task on P2P's Pass 1, but as the bug states, it blocks far more than just the specific target.

Most MSBuild task target (full) Build or to get outputs, so the entire project has to finish before returning and the build's outputs only relevant at the end of the build. So single pass build won't hit this issue.

ak-slongchamps commented 12 months ago

Adding my voice to this issue: our use case is that we have a couple of compilation units that are quite heavy to build (typically by orders of magnitude compared to the average) in base libraries. Those compilation units collapse a normally fully parallel build to a single-threaded, single project build point as a base library finishes to compile those heavy compilation units. Making those compilation units lighter by splitting them to increase parallelization usually means duplicating a lot of frontend work that ends up in a slower build overall, so this is not a fix.

Lots of work could be accomplish during that time by starting the build phases of downstream libraries in the solution, for which AllowParallelCompileInReferencedProjects is the adequate way to express that. This would be a huge time saver for us.