search

dotnet / msbuild

The Microsoft Build Engine (MSBuild) is the build platform for .NET and Visual Studio.
https://docs.microsoft.com/visualstudio/msbuild/msbuild
MIT License
5.17k stars 1.34k forks source link

[Performance]: dotnet/runtime tests are bottlenecked by MSBuild performance #10021

Open agocke opened 3 months ago

agocke commented 3 months ago

Issue Description

The dotnet/runtime tests consist of thousands of tests in the src/tests tree. In the lab these tests take upwards of 20 minutes to build.

However, it looks like most of this time is in MSBuild. When I build the first fraction of these tests (1100 tests) locally, it takes MSBuild 6:50s.

If I run the same projects through Roslyn's "Replay" runner, which runs the csc commands from the binlog, this takes 1:39s.

That's a roughly 5x speedup.

The overall cost here is significant complexity in the runtime build. The other piece of important info is that the libraries tests, take only a few minutes to build. That means that if we were to build the entire test tree together we would delay starting the libraries tests by > 20 minutes. To counteract this we make the build substantially more complicated by splitting out a separate job in our pipeline just to build the runtime tests. If the overall build time were significantly reduced, we could remove a lot of complexity and delay in our CI testing.

Steps to Reproduce

See dotnet/runtime src/tests tree. The command line I used to build was src/tests/build.sh allTargets skipnative skipgeneratelayout skiptestwrappers checked x64 /p:LibrariesConfiguration=Release -bl

Data

MSBuild 6:50s. Csc 1:39s.

Analysis

No response

Versions & Configurations

No response

Regression

Regression Details

No response

agocke commented 3 months ago

fyi @rainersigwald

danmoseley commented 3 months ago

I thought the North Star here was far fewer assemblies?

agocke commented 3 months ago

You mean fewer projects? We've already performed a significant amount of consolidation. We can continue to shrink this down over time, but it will still likely be a lot of projects. And it doesn't look like csc has a problem with this scale.

agocke commented 3 months ago

The other reason this came up is that the 1ES pipeline work revealed how extremely complicated the runtime pipeline is. It's easily the largest pipeline in dotnet, exceeding standard AzDO rate limits by two orders of magnitude. The runtime tests currently represent an unresolvable complication in our AzDO pipelines wherein we would like to remove the extra job + upload + join, but can't do so if it would regress time-to-test-results by 20 min.

danmoseley commented 3 months ago

You mean fewer projects? We've already performed a significant amount of consolidation. We can continue to shrink this down over time, but it will still likely be a lot of projects.

Right. I'm not defending MSBuild. But why are more than say O(100) assemblies needed? I'm just curious, not suggesting I have any special insight.

agocke commented 3 months ago

I believe it's a mix of: Native AOT tests can't be mixed because more code has trimming side effects, some tests contaminate the process such that it can't be reused, some simply have code patterns that can't easily be combined, and finally test consolidation still ends up with a fair amount of manual effort and we can only commit so much per unit time.

rokonec commented 1 month ago

I am not saying there is no room for improvement, but MSBuild does lots more than csc tasks. Most of the work spent by MSBuild outside of csc task is to ensure it is up-to-date and consistent. For that have IO operations has to run, mostly involving checking file existence, timestamps, globing, etc... to detect changes in file system.