Slow build with msbuild and dotnet cli

[NinoFloris]

Ok so we migrated over a few weeks ago but I'm really getting annoyed by the lack of incremental build on rc4. Debug times take 13 seconds from f5 to debugger start because dotnet build lacks any and all smarts to see that only the top level project changed so it just rebuilds all projects again.

I don't have any experience with msbuild and I'd expect at least this to just work out of the box. It's even not working for the simplest case

• dotnet new mvc
• dotnet restore
• dotnet build first time builds nicely
• dotnet build changed nothing, starts all over again wasting 3 seconds of my time.

.NET Command Line Tools (1.0.0-rc4-004849)

Product Information:
 Version:            1.0.0-rc4-004849
 Commit SHA-1 hash:  7dcefb5076

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/1.0.0-rc4-004849

[davkean]

Some recent progress, for example, is: https://github.com/dotnet/standard/issues/442 and https://github.com/Microsoft/msbuild/issues/2392.

[firelizzard18]

I created dotnet/cli#7482 in case I was experiencing anything distinct from the issues detailed here. As far as performance, I'm exceeding a minute for large builds (the full dependency tree for my top-level projects includes 50+ project references, not to mention packages and standard libraries).

What's killing my builds is ResolveProjectReferences.

[Jetski5822]

Agreed - Build Orchard which has around 132 project is taking around 5 minutes on my i7 Surface 3 Pro... The resolving of project references just seems to last forever and never stops.

[NinoFloris]

Improvements are coming with msbuild 15.5 and should relieve some of the bigger issues. That's correct right @davkean ?

[firelizzard18]

@NinoFloris When can I expect to have msbuild 15.5?

[davkean]

We're focusing on performance for the next couple of updates, you can see some of the things we've already tackled for 15.5/2.1 here: https://github.com/dotnet/project-system/issues/2789.

For Visual Studio 15.5/CLI 2.1, we've been mainly focused on improving evaluation time. Slow evaluation time is what what results in large build times for the ResolveProjectReferences/_GetProjectReferenceTargetFrameworkProperties targets, results in delays when adding/removing files from a project and results in UI delays when referencing .NET Standard/.NET Core projects from .NET Framework projects. We saw a 30% - 60% build time improvement for some large solutions from this work. Make note, small projects or projects without project references won't see much improvement from this, that will be focused in Visual Studio 15.6/CLI 2.2.

For Visual Studio 15.6/CLI 2.2, we'll be focusing on improving what we call the inner loop; basically all the things you do over and over again while developing an application. This includes, but is not limited to, looking at both up-to-date and out-of-date CLI builds/restores, the dreaded ResolveAssemblyReference target (which is significantly worse in .NET Core) and improving the startup time of the CLI.

[plaisted]

I have what I consider a typical small application. Two solutions with ~20 projects including test projects.

On my CI server I essentially run "dotnet test" on all the test projects. This takes quite a long time even though the tests run quickly. The real problem seems to be that on each test project rebuilds it's dependencies even if another project has already built them. So if I have 4 test projects that depend on the same 5 libraries (projects) this results in 24 projects being built instead of just 9 (5 libraries get built 4 times each).

Is there a way to get around this? I've thought about manually building each project and walking the dependency tree (with "dotnet build --no-dependencies") and then doing "dotnet test --no-build" to solve this but that would require me writing something to walk the dependency tree and coordinate it or "hard code" the build steps and keep that in sync with the projects as the dependency graph changes. I feel like I'm having to rewrite the whole build system to make up for the CLI's short comings.

Seems like either:

• Having rebuilds for dependent projects be quick if they have been already built (this issue), or,
• Adding a flag like "--no-dependencies-if-exists" which only builds them if the DLL doesn't already exist would solve most of the issues here.

Maybe (hopefully) I'm missing something and this is already resolved someway else in the CLI.

[dasMulli]

@davkean thanks for that update! I was wondering about one particular feature: Reference assembly support. MSBuild/Roslyn already support this and my last info was that at least in 15.3 the classic project system didn't yet support it. Is that changing with 15.5+? Something preventing the SDK to opt into this feature by default? (.net core/standard projects are already <Deterministic>True</…>).

[tverboon]

@plaisted

You could create an MSBuild file for testing. Maybe this helps: https://github.com/Microsoft/vstest/issues/411#issuecomment-322383204

[plaisted]

@tverboon

Thanks for the suggestion, that seems to fix the repeated building issue, knocked the time for a full testing run in half approximately. Would be nice to have this sort of functionality through the CLI.

[NinoFloris]

We actually use dotnet vstest #dllpath to execute build free tests, this command also allows you to add multiple dll paths to run different assemblies sequentially

On Fri, Nov 10, 2017, 20:39 Michael Plaisted notifications@github.com wrote:

@tverboon https://github.com/tverboon

Thanks for the suggestion, that seems to fix the repeated building issue, knocked the time for a for testing run in half approximately.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dotnet/cli/issues/5918#issuecomment-343566742, or mute the thread https://github.com/notifications/unsubscribe-auth/AEBfudmuiYDZ2mpUsztvXFc3xJWcFPj_ks5s1KZ-gaJpZM4MTJlk .

[dasMulli]

Incremental build times are improving using the 15.5 msbuild/cli version and the current latest bits (containing additional improvements). Using https://github.com/dasMulli/cli-incremental-perf-testbed, unscientific measurements for incremental builds (macOS 10.13.1, time dotnet build --no-restore) are:

run / version	2.0.3	2.1.3-preview-007232	2.2.0-preview1-007736
initial	1m17s	56s	41s
1st inremental	55s	36s	22s
2nd incremental	1m3s	37s	23s
3rd incremental	52s	34s	22s

[odinnix]

When is 15.5 expected to release for Cli? I was hoping with VS 15.5.

[nguerrera]

@dcarl1 The 2.1.2 that released with VS 15.5 should have the same perf as that middle column.

[nguerrera]

(Downloads of 2.1.2 for other platforms aren't up yet, but will be soon.)

[odinnix]

@nguerrera for CLI tooling (dotnet run)? The newest version I can find is 2.0.3 at https://github.com/dotnet/cli/releases

[NinoFloris]

@nguerrera 'soon', great... I'm not really happy I cannot even find a beta build of it anywhere for the other platforms. Many .net core users are not using the standard windows + vs stack which usually is why they're a .net core user at all. Your logic to push these things first to VS is flawed and seems to again lack the willingness to keep the other platforms on the same level of quality and support. Sorry to say but people not using vs are hit the hardest by slow build times as they don't have vs's up-to-date checks etc, for us it's rebuild all day everyday. We'd like to finally see some improvement.

[livarcocc]

cc @leecow

[giggio]

I got 2.1.2, I guess it came with VS 15.5, because I have not installed it manually. And yes, it is bad that the latest on the download page shows 2.0.3. It should be up there by now.

[giggio]

I just checked and the releases page here on Github is also not showing even a tag for 2.1.2.

[dasMulli]

Just to make sure the feature is known:

When using my "testbed" linked above with "lib1, referenced by lib2, which is in turn referenced by lib3, which is in turn" ... up to lib20, changing an implementation detail of lib1 causes the build time to be the same as the initial build.

However when adding this to the Directory.Build.props file (or added to every csproj file):

  <PropertyGroup>
    <ProduceReferenceAssembly>true</ProduceReferenceAssembly>
  </PropertyGroup>

The build time is only 1-2 seconds higher during incremental builds (using 2.2.0 CI build, but the feature should already be available in 2.0.0+). Only when the public interface of any assembly is changed (e.g. signature of public methods, types) the build time is as long as the initial build again.

[nguerrera]

@AndyGerlicher @jaredpar How would you feel about turning that on by default for sdk projects?

Failing that, @Pilchie, could we get some nice UI and maybe an info bar to turn it on?

[Pilchie]

Also tagging @jcouv and @rainersigwald . I thought we had turned it on for SDK projects. If not, what are the drawbacks? That it takes longer to produce binaries that are not referenced? Could we turn it on by default for libraries, but not programs or tests?

[AndyGerlicher]

I think the fast-up-to-date check was the last thing holding back from general usage. I don't remember the details, but I think it couldn't statically be known whether or not to rebuild. So it was either always rebuilding or not in certain cases when it should. If we feel comfortable with the state of that with the SDK projects it should definitely be turned on.

[NinoFloris]

@Pilchie Beware though, if I turn this on for all projects in a mixed solution (C# + F#) I get errors where it can't find the ref assembly for the F# projects.

[jeromelaban]

Should this issue be fixed beforehand ? https://github.com/Microsoft/msbuild/issues/2431. With VS15.5, incrementally updated assemblies are not picked up the head projects, where the up to date check indeed seems to be too agressive.

[davkean]

Folks, we're still making some pretty good progress - this change https://github.com/NuGet/NuGet.Client/pull/1866 significantly reduces the overhead of implicit restore. We saw up-to-date dotnet build for https://github.com/OrchardCMS/OrchardCore reduce by about 25%.

[FransBouma]

Is there any info about whether 'dotnet build' performs poorly inside a VM (e.g. a VMWare VM with 6 virtual CPUs) ? I have a large .sln with 47 multitargeting csproj files (targeting .net 4.5.2 and .netstandard2.0) (the projects used by our unit tests, they're all generated).

I do in a cmd file:

@Echo off

set destinationRoot="\SomeFolder\"

pushd.
dotnet build -f netstandard2.0 --no-restore -o %destinationRoot%\netstandard2.0 AllUnitTestsSupportProjects.sln 
dotnet build -f net452 --no-restore -o  %destinationRoot%\net452 AllUnitTestsSupportProjects.sln
popd

this needs sometimes 1m22 or 2m23 or thereabout (I don't know why this fluctuates, nothing else is going on on the machine nor VM). Quite slow. When I do on the commandline: msbuild AllUnitTestsSupportProjects.sln /v:m /t:rebuild after a clean, it takes msbuild 22 seconds to build everything. When I do: dotnet build --no-incremental --no-restore AllUnitTestsSupportProjects.sln it takes 2m12 seconds.

This is inside the VM, VMWare, windows 8.1 x64, visual studio 15.5.5, dotnet 2.1.4. msbuild version: Microsoft (R) Build Engine version 15.5.180.51428 for .NET Framework.

So I copied literally all files in this solution from the VM to the host running the VM. On it vs 2017 15.5.6 (the VM's vs installer apparently can't see an update, the one on the host os did, that aside), dotnet 2.1.4, msbuild version Microsoft (R) Build Engine version 15.5.180.51428 for .NET Core, and the exact same .cmd file takes 41 seconds. Now, VMWare might be slow at times, but not this slow. Also the massive discrepancy between MSBuild on the guest vs. dotnet build isn't normal IMHO.

So my question to you is: what can I do to dig up some information to track down the root cause of this? dotnet build is way slower than msbuild it seems, but even then, it shouldn't be that much of a difference between host and guest on the same hw.

TIA

[mikeharder]

@FransBouma: We haven't seen significant perf differences between a Hyper-V VM and the host. My first guess would be a difference in disk perf. If you can share your solution, I can measure the perf on a physical machine and a Hyper-V VM to see how they compare.

You can also try a daily build of 2.1.300-preview1 which has many performance improvements to .NET Core build. I would expect the relative performance to be the same between the VM and host, but both should be significantly faster.

[FransBouma]

@mikeharder I can't share this code sadly (it's not open source), however I've tried the 2.1.300 preview1 and it solved things :) The total build time is now ~22 seconds (retested) on the guest (VM), so I see the problem I had solved, as the upcoming release contains the fix. No further action required :) (On the host, the 2.1.300 preview1 takes 15 seconds, with 8 core parallel build). Nice progress! :)

[andycmaj]

have a question about build times that might be related to this...

I am working with a large solution with a UI app that delegates Areas (controllers/commands/models/etc.) to a bunch of smaller projects.

I was trying to consolidate all the Area projects (an arbitrary partition, dependency/design-wise) into the main UI project.

example:

root
  - UI.csproj (68 .cs files)
  - Area.Foo.csproj
  - Area.Bar.csproj

| v

root
  - UI.csproj (537 .cs files)
    - Area.Foo code
    - Area.Bar code

When i built the Before state, after making a change to UI.csproj, i hit about 43 second builds (with 68 files).

After consolidating, builds now take about 3-4 minutes (with 537 .cs files).

I understand why performance would be worse here. Seems like incremental build heuristic can make easier decisions about which src files it doesn't have to re-build if changes are localized to a smaller project that has clearly defined downstream consumer projects.

This make sense, but my original expectation is that i'd save time building/walking/comprehending the dependency tree and transitive package/project dependencies.

In any event... would it be considered a "best practice" or a "recommendation" to split out slow-building projects into lots of smaller partitions for the sake of taking advantage of incremental builds?

[davkean]

It's hard to know without seeing where the time is spent. Do a diagnostic builds of both states and share out the performance sections at the bottom of the log. I suspect csc (compile) is probably the bottle neck.

[NinoFloris]

@davkean any news on reference assembly support for local projects?

So we only rebuild full chain on api changes, or too many issues?

[Pilchie]

@rainersigwald for this one.

[rainersigwald]

@NinoFloris You can opt in to reference assembly support today. There are some docs here but the easiest way to enable it is to set

<PropertyGroup>
  <ProduceReferenceAssembly>true</ProduceReferenceAssembly>
</PropertyGroup>

in a Directory.Build.props file that applies to your projects.

reads upward

Oh, I see you've already tried that and were seeing problems with F#? Is that the problem you're having?

[NullVoxPopuli]

I'm having problems with this and F# as well. it's like it lost track of all my nuget packages

[nguerrera]

@NullVoxPopuli That sounds like a different, more specific issue. Please file it as a new issue and include the steps you take to reproduce the problem. Thanks.

[livarcocc]

Closing this issue as we have done a series of perf improvements for SDK 2.1.300 and while we will continue improving on it, we are not planning any other changes in this area for this release.

[davkean]

As a heads up, more improvements have been made in Preview 2 - https://blogs.msdn.microsoft.com/dotnet/2018/04/11/announcing-net-core-2-1-preview-2/.

[forki]

That sounds good.

[nn-afly]

Dramatic improvement here. Excellent work 👍

[ghost]

Unfortunately it is still slow on some projects. For example on OrchardCore CMS. The main project has internal dependencies which are every time recompiled and it takes like 6 minutes every build.

You can check it out here: https://github.com/OrchardCMS/OrchardCore

Tested with 2.1.300 on Linux.

[sebastienros]

@adamos101 I can't see this behavior locally with Orchard Core. Even on our CI which uses lower end machine it's faster than that. Also I assume the numbers the team is showing is using Orchard Core for the large project test.

Do you know which dependencies are triggering the issue you are seeing?

[mikeharder]

@adamos101: OrchardCore is actually part of our build perf test suite, though we use a fixed commit SHA (currently https://github.com/OrchardCMS/OrchardCore/tree/21ab3a56d0b93bb5c89285d18faa7b788354c4c0 from Feb 8 2018) rather than the tip of dev.

Here are the warm build times we measure on an Azure DS4_v2 (8-core, SSD) VM running Ubuntu 16.04:

Build Type	Time (s)
Full	55.650
Incremental	22.510

If you are seeing significantly different results (like 6 minutes for an incremental build), please let us know how to reproduce it. I will also update the commit SHA we test to the tip of dev, to see if a recent change in the OrchardCore repo might be causing this.

[ghost]

@mikeharder sorry for the hassle, forget it. It doesn't rebuild the DLLs, they have the same timestamp. It just takes long time to go thru all the project and it is slow on my machine. I think I will need some faster quad core for that. That's why I was thinking it is rebuilding it because on my machine it took 6 minutes but it wasnt rebuilding at all. False alarm :-)

[mikeharder]

@adamos101: We are concerned about slow builds regardless of the root cause. Would you mind sharing the specs of your machine (CPU and disk), the exact command you are running, and how long it takes? I can try to repro on an Azure VM similar to your machine.

[mikeharder]

I am seeing a significant increase in OrchardCore incremental build time between Feb 8 and today:

Commit	Incremental Build Time (s)
5b306d0 (2/8/18)	22.510
e75d7c3 (6/13/18)	37.160

We will investigate this difference, but it might be expected based on the changes to OrchardCore between these dates.