search

aspnet / Benchmarks

Benchmarks for ASP.NET Core
Other
1.17k stars 238 forks source link

HttpClient Benchmarks: server build crashes on 9.0 #2018

Open CarnaViire opened 3 weeks ago

CarnaViire commented 3 weeks ago

Follow-up from https://github.com/aspnet/Benchmarks/issues/2006

Server app crashes the build job if 9.0 TFM is requested, but 8.0 works. Client app builds and works properly on 9.0 though.

--profile intel-lin-app --profile amd-lin2-load, --server.framework net9.0

Logs ```bash ➜ ~ crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile intel-lin-app --profile amd-lin2-load --variable useHttpMessageInvoker=true --variable httpVersion=1.1 --variable useHttps=true --variable numberOfHttpClients=1 --variable concurrencyPerHttpClient=10 --variable responseSize=256 --client.framework net9.0 --server.framework net9.0 --client.options.collectCounters true [08:45:29.594] Running session 'aba271939cbb463b991183f5a0ffcd3b' with description '' [08:45:29.990] Starting job 'server' ... [08:45:30.171] Submitted job: http://asp-citrine-amd2:5001/jobs/151 [08:45:30.531] 'server' has been selected by the server ... [08:45:30.704] 'server' is now building ... http://asp-citrine-amd2:5001/jobs/151/buildlog [08:45:33.591] 'server' failed on agent, stopping... An unexpected error occurred while building the job: One or more errors occurred. (Object reference not set to an instance of an object.) [08:45:33.946] Job has failed, interrupting benchmarks ... [08:45:33.949] Stopping job 'server' ... [08:45:35.483] Deleting job 'server' ... ```

Not able to repro on a local agent (on Windows, Crank Agent version 0.2.0-alpha.24379.1+a59cbdbe3f8fc7aef08c17652cc2cef6136464ca):

--profile local, --server.framework net9.0

Logs ```bash ➜ ~ ✗ crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile local --variable useHttpMessageInvoker=true --variable httpVersion=1.1 --variable useHttps=true --variable numberOfHttpClients=1 --variable concurrencyPerHttpClient=10 --variable responseSize=256 --client.framework net9.0 --server.framework net9.0 --client.options.collectCounters true [08:57:03.867] Running session 'fe2a0f689db84a6cb11ba8ae7061a6b4' with description '' [08:57:03.907] Starting job 'server' ... [08:57:04.021] Submitted job: http://localhost:5010/jobs/1 [08:57:08.098] 'server' has been selected by the server ... [08:57:08.102] 'server' is now building ... http://localhost:5010/jobs/1/buildlog [08:59:14.528] 'server' is running ... http://localhost:5010/jobs/1/output [08:59:14.537] Starting job 'client' ... [08:59:14.540] Submitted job: http://localhost:5010/jobs/2 [08:59:16.566] 'client' has been selected by the server ... [08:59:16.567] 'client' is now building ... http://localhost:5010/jobs/2/buildlog [08:59:26.686] 'client' is running ... http://localhost:5010/jobs/2/output [09:00:00.997] Stopping job 'client' ... [09:00:02.051] Deleting job 'client' ... [09:00:02.054] Stopping job 'server' ... [09:00:04.086] Deleting job 'server' ... | server | | | ----------------------- | -------------------------------- | | Max CPU Usage (%) | 47 | | Max Cores usage (%) | 560 | | Max Working Set (MB) | 75 | | Max Private Memory (MB) | 32 | | Build Time (ms) | 45,501 | | Start Time (ms) | 5,682 | | Published Size (KB) | 101,208 | | Symbols Size (KB) | 23 | | .NET Core SDK Version | 9.0.100-rc.1.24415.1 | | ASP.NET Core Version | 9.0.0-rc.1.24412.15+abbd67b97144 | | .NET Runtime Version | 9.0.0-rc.1.24413.14+605ff6f64851 | | client | | | --------------------------------------- | -------------------------------- | | Max CPU Usage (%) | 33 | | Max Cores usage (%) | 391 | | Max Working Set (MB) | 61 | | Max Private Memory (MB) | 30 | | Build Time (ms) | 3,688 | | Start Time (ms) | 1,162 | | Published Size (KB) | 74,917 | | Symbols Size (KB) | 19 | | .NET Core SDK Version | 9.0.100-rc.1.24415.1 | | ASP.NET Core Version | 9.0.0-rc.1.24412.15+abbd67b97144 | | .NET Runtime Version | 9.0.0-rc.1.24413.14+605ff6f64851 | | Processor Count | 12 | | First request duration (ms) | 794 | | Requests | 657,722 | | Bad Status Code Requests | 0 | | Exceptions | 0 | | Mean RPS | 43,805 | | Max CPU Usage (%) | 31 | | Max Working Set (MB) | 63 | | Max GC Heap Size (MB) | 5 | | Size of committed memory by the GC (MB) | 6 | | Max Number of Gen 0 GCs / sec | 26.00 | | Max Number of Gen 1 GCs / sec | 1.00 | | Max Number of Gen 2 GCs / sec | 0.00 | | Max Gen 0 GC Budget (MB) | 4 | | Max Time in GC (%) | 5.00 | | Max Gen 0 Size (B) | 1,702,744 | | Max Gen 1 Size (B) | 819,664 | | Max Gen 2 Size (B) | 1,574,656 | | Max LOH Size (B) | 0 | | Max POH Size (B) | 24,528 | | Max Allocation Rate (B/sec) | 65,893,200 | | Max GC Heap Fragmentation (%) | 4,135% | | # of Assemblies Loaded | 42 | | Max Exceptions (#/s) | 2 | | Max Lock Contention (#/s) | 5 | | Max ThreadPool Threads Count | 20 | | Max ThreadPool Queue Length | 1 | | Max ThreadPool Items (#/s) | 88,857 | | Max Active Timers | 2 | | IL Jitted (B) | 282,332 | | Methods Jitted | 3,147 | ```
Crank Agent logs ```bash ➜ crank git:(main) crank-agent [20:56:53 INF] Crank Agent version 0.2.0-alpha.24379.1+a59cbdbe3f8fc7aef08c17652cc2cef6136464ca [20:56:53 INF] Starting agent on http://*:5010... .... [20:57:08 INF] Starting job 'server' (1) .... [20:57:10 INF] Installing dotnet runtimes and sdk [20:57:10 INF] Specific target framework: 'net9.0' [20:57:10 INF] Patching project file 'C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\5x21ruwe.2vt\src\BenchmarksApps\HttpClientBenchmarks\Servers\Kestrel\Kestrel.csproj' [20:57:12 INF] Runtime: 9.0.0-rc.1.24413.14 (Latest - From 9.0 SDK) [20:57:12 INF] SDK: 9.0.100-rc.1.24415.1 (Latest - From Product Commit) [20:57:13 INF] ASP.NET: 9.0.0-rc.1.24412.15 (Latest - From 9.0 SDK) [20:57:13 INF] Creating custom global.json [20:57:13 INF] Desktop: 9.0.0-preview.7.24405.2 (Latest) [20:57:13 INF] Installing SDK '9.0.100-rc.1.24415.1' ... [20:57:13 INF] Checking package: https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-rc.1.24415.1/dotnet-sdk-9.0.100-rc.1.24415.1-win-x64.zip [20:58:16 INF] Installing Runtime '9.0.0-rc.1.24413.14' ... [20:58:16 INF] Checking package: https://dotnetbuilds.azureedge.net/public/Runtime/9.0.0-rc.1.24413.14/dotnet-runtime-9.0.0-rc.1.24413.14-win-x64.zip [20:58:18 INF] Installing Desktop runtime '9.0.0-preview.7.24405.2' ... [20:58:18 INF] Checking package: https://dotnetbuilds.azureedge.net/public/Runtime/9.0.0-preview.7.24405.2/windowsdesktop-runtime-9.0.0-preview.7.24405.2-win-x64.zip [20:58:19 INF] Checking package: https://dotnetcli.azureedge.net/dotnet/Runtime/9.0.0-preview.7.24405.2/windowsdesktop-runtime-9.0.0-preview.7.24405.2-win-x64.zip [20:58:19 INF] Installing ASP.NET runtime '9.0.0-rc.1.24412.15' ... [20:58:19 INF] Checking package: https://dotnetbuilds.azureedge.net/public/aspnetcore/Runtime/9.0.0-rc.1.24412.15/aspnetcore-runtime-9.0.0-rc.1.24412.15-win-x64.zip [20:58:22 INF] Working directory: C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\5x21ruwe.2vt\src\BenchmarksApps\HttpClientBenchmarks\Servers\Kestrel [20:58:22 INF] Command line: C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\bqtroxbz.0w0\dotnet.exe publish Kestrel.csproj -c Release -o C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\5x21ruwe.2vt\src\BenchmarksApps\HttpClientBenchmarks\Servers\Kestrel\published /p:MicrosoftNETCoreAppPackageVersion=9.0.0-rc.1.24413.14 /p:MicrosoftAspNetCoreAppPackageVersion=9.0.0-rc.1.24412.15 /p:GenerateErrorForMissingTargetingPacks=false /p:RestoreNoCache=true /p:MicrosoftWindowsDesktopAppPackageVersion=9.0.0-rc.1.24414.1 /p:MicrosoftNETPlatformLibrary=Microsoft.NETCore.App --framework net9.0 --self-contained -r win-x64 [20:59:07 INF] Application published successfully in 45501.9321 ms .... [20:59:18 INF] Starting job 'client' (2) .... [20:59:19 INF] Installing dotnet runtimes and sdk [20:59:19 INF] Specific target framework: 'net9.0' [20:59:19 INF] Patching project file 'C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\5vkazsrp.a35\src\BenchmarksApps\HttpClientBenchmarks\Clients\HttpClient\HttpClient.csproj' [20:59:21 INF] Runtime: 9.0.0-rc.1.24413.14 (Latest - From 9.0 SDK) [20:59:21 INF] SDK: 9.0.100-rc.1.24415.1 (Latest - From Product Commit) [20:59:21 INF] ASP.NET: 9.0.0-rc.1.24412.15 (Latest - From 9.0 SDK) [20:59:21 INF] Creating custom global.json [20:59:21 INF] Desktop: 9.0.0-preview.7.24405.2 (Latest) [20:59:21 INF] Working directory: C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\5vkazsrp.a35\src\BenchmarksApps\HttpClientBenchmarks\Clients\HttpClient [20:59:21 INF] Command line: C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\bqtroxbz.0w0\dotnet.exe publish HttpClient.csproj -c Release -o C:\Users\knatalia\AppData\Local\Temp\benchmarks-agent\benchmarks-server-30908\5vkazsrp.a35\src\BenchmarksApps\HttpClientBenchmarks\Clients\HttpClient\published /p:MicrosoftNETCoreAppPackageVersion=9.0.0-rc.1.24413.14 /p:MicrosoftAspNetCoreAppPackageVersion=9.0.0-rc.1.24412.15 /p:GenerateErrorForMissingTargetingPacks=false /p:RestoreNoCache=true /p:MicrosoftWindowsDesktopAppPackageVersion=9.0.0-rc.1.24414.1 /p:MicrosoftNETPlatformLibrary=Microsoft.NETCore.App --framework net9.0 --self-contained -r win-x64 [20:59:25 INF] Application published successfully in 3688.6041 ms .... ```
sebastienros commented 3 weeks ago

Was able to reproduce with the command line you provided. The amd2 machine was actually not up-to-date and didn't have the fix that solved the sdks resolution. It's working fine now.