Building an ASP.NET project 2-10x slower with .NET 9 vs .NET 8

nwoolls commented 1 week ago

Describe the bug

Building ASP.NET MVC applications with .NET 9 is 2x slower than .NET 8 for basic applications and 10x slower for applications that utilize several client-side libraries (e.g. with LibMan).

To Reproduce

See the following GitHub repository to reproduce: https://github.com/nwoolls/spl-dotnet-9-slow-build

The applications in this project were created with dotnet new mvc. The projects in the simple folder stop at that. The projects in the advanced folder have a handful of client-side libraries that are installed using the LibMan CLI.

Running benchmark.js from the repository yields:

Benchmark results: Simple MVC .NET 8 (avg. seconds): 1.05 Simple MVC .NET 9 (avg. seconds): 2.08 Advanced MVC .NET 8 (avg. seconds): 3.48 Advanced MVC .NET 9 (avg. seconds): 30.96

Exceptions (if any)

Running dotnet clean before dotnet build results in dramatically faster build times, even though the client-side libraries are restored by LibMan prior to the BeforeBuild MSBuild target.

Benchmark results: Simple MVC .NET 8 (avg. seconds): 1.15 Simple MVC .NET 9 (avg. seconds): 2.06 Advanced MVC .NET 8 (avg. seconds): 2.21 Advanced MVC .NET 9 (avg. seconds): 2.42

Further technical details

Include the output of dotnet —info

dotnet --info
.NET SDK:
 Version:           8.0.400
 Commit:            36fe6dda56
 Workload version:  8.0.400-manifests.27c4fe60
 MSBuild version:   17.11.3+0c8610977

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  14.6
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.400/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
There are no installed workloads to display.

Host:
  Version:      9.0.0-rc.1.24431.7
  Architecture: arm64
  Commit:       static

.NET SDKs installed:
  6.0.425 [/usr/local/share/dotnet/sdk]
  8.0.400 [/usr/local/share/dotnet/sdk]
  9.0.100-preview.7.24407.12 [/usr/local/share/dotnet/sdk]
  9.0.100-rc.1.24452.12 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.33 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-preview.7.24406.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-rc.1.24452.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.33 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-preview.7.24405.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-rc.1.24431.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  /Users/nwoolls/src/dotnet/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

The IDE (VS / VS Code/ VS4Mac) you're running on, and its version

None

baronfel commented 6 days ago

I don't repro the slowdowns - can you do this again and get binlogs for each run? The in every case, the step that took the longest was the BeforeBuildTasks call of the libman restore tool - this takes several seconds on every build and is not at all designed in the way a good MSBuild integration should be (i.e. tracking incrementality, inputs and outputs, doing some detection to see if the command needs to be run).

It's my strong suspicion that libman is the cause of your delays, and any fixes should be referred to the owners of that tool.

nwoolls commented 6 days ago

@baronfel will do now and post my findings. FWIW that target is not what takes appreciable time for me. Here it's ResolveProjectStaticWebAssets and ResolveBuildCompressedStaticWebAssets - each taking seemingly 10s of seconds.

baronfel commented 6 days ago

in that case cc @javiercn for static web assets delays

nwoolls commented 6 days ago

@baronfel also - not disagreeing per se, but if this were a libman issue I don't see why it wouldn't exist in .NET 8, or why a dotnet clean first would seem to dramatically decrease build times.

baronfel commented 6 days ago

I can only report my findings, I don't know what to tell you otherwise. I am on Windows though which may have different performance characteristics from your macOS host! In any case, binlogs from your repo will be the most accurate thing for us to work from.

nwoolls commented 6 days ago

@baronfel MSBuild binary log files generated and included w/ the repo - e.g.:

https://github.com/nwoolls/spl-dotnet-9-slow-build/blob/main/advanced/mvc-dotnet-9/msbuild.binlog

baronfel commented 6 days ago

Thanks, those really helped! They did confirm that the new static web assets content type handling can be really quite slow:

This is a pretty big slowdown for several hundred assets, I expect this will have very large penalties in real-world applications.

nwoolls commented 6 days ago

I expect this will have very large penalties in real-world applications.

It does 😆.

I apologize for not reporting sooner — this is what kept me away from earlier preview releases. And why I came back for the RC to see if it was fixed.

MackinnonBuck commented 3 days ago

We did have a perf fix for DefineStaticWebAssetEndpoints in https://github.com/dotnet/sdk/pull/43099, which didn't make the cut for RC1. However, I just tried the repro on the latest nightly build and still see the perf issues described in this issue. I collected a trace, and it seems most of the time spent in the static web assets pipeline is on JSON serialization and file system globbing, but @javiercn would know better what the best way to optimize this is.

dotnet / sdk