search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.6k stars 971 forks source link

[DOC] Wasm doc needs update #1818

Open naricc opened 3 years ago

naricc commented 3 years ago

This document and assocaited source code need updating, since --wasmMainJS and --customruntimepack options are no longer needed, and the --runtimeSrcDir argument is now required: https://github.com/dotnet/BenchmarkDotNet/blob/master/docs/articles/samples/IntroWasm.md/#L1

I intend to take on this task; creating the issue so I can keep track of things.

erikthysell commented 2 years ago

Thanks for the initiative @naricc . I would love to have some inspiration on how to compare different versions of wasm runtimes (i.e. .net 5 vs .net 6 wasm, and perhaps even aot wasm)

SamMonoRT commented 2 years ago

@radical - sending this your way, as many changes have been done for wasm related pipeline which needs documentation.

adamsitnik commented 2 years ago

I wanted to benchmark WASM AOT today to unblock https://github.com/dotnet/runtime/pull/73556#issuecomment-1208101529

  1. We have zero docs in both dotnet/BenchmarkDotNet and dotnet/performance that describe how to do it.
  2. I have read the code and got the impression that I need to pass following args:
--runtimes wasm --AOTCompilerMode wasm --wasmDataDir /home/adam/projects/runtime/src/mono/wasm/test-main.js

Which gave me following errors:

/home/adam/projects/performance/tools/dotnet/x64/sdk/7.0.100-rc.1.22407.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.ImportWorkloads.targets(38,5): error NETSDK1147: To build this project, the following workloads must be installed: wasm-tools [/home/adam/projects/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/27456a6c-b7a5-4af8-8816-6675f6cb0439/BenchmarkDotNet.Autogenerated.csproj
/home/adam/projects/performance/tools/dotnet/x64/sdk/7.0.100-rc.1.22407.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.ImportWorkloads.targets(38,5): error NETSDK1147: To install these workloads, run the following command: dotnet workload restore [/home/adam/projects/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/27456a6c-b7a5-4af8-8816-6675f6cb0439/BenchmarkDotNet.Autogenerated.csproj]

I can see that BDN tries to perform dotnet restore while it should perform dotnet workload restore.

I've tried to install the workload manually:

cd /home/adam/projects/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/27456a6c-b7a5-4af8-8816-6675f6cb0439/
/home/adam/projects/performance/tools/dotnet/x64/dotnet workload restore --packages "/home/adam/projects/performance/artifacts/packages" /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true --project ./BenchmarkDotNet.Autogenerated.csproj

And got new failures:

Skipping NuGet package signature verification.
Installing workload manifest microsoft.net.sdk.maui version 6.0.449…
Installing workload manifest microsoft.net.workload.mono.toolchain version 7.0.0-rc.1.22408.1…
Installing workload manifest microsoft.net.workload.emscripten version 7.0.0-rc.1.22405.3…
Installing pack Microsoft.NET.Runtime.WebAssembly.Sdk version 7.0.0-rc.1.22408.1...
Writing workload pack installation record for Microsoft.NET.Runtime.WebAssembly.Sdk version 7.0.0-rc.1.22408.1...
Installing pack Microsoft.NETCore.App.Runtime.Mono.browser-wasm version 7.0.0-rc.1.22408.1...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.browser-wasm version 7.0.0-rc.1.22408.1...
Installing pack Microsoft.NETCore.App.Runtime.AOT.linux-x64.Cross.browser-wasm version 7.0.0-rc.1.22408.1...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.browser-wasm version 7.0.0-rc.1.22408.1...
Installing pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 7.0.0-rc.1.22408.1...
Writing workload pack installation record for Microsoft.NET.Runtime.MonoAOTCompiler.Task version 7.0.0-rc.1.22408.1...
Installing pack Microsoft.NET.Runtime.MonoTargets.Sdk version 7.0.0-rc.1.22408.1...
Writing workload pack installation record for Microsoft.NET.Runtime.MonoTargets.Sdk version 7.0.0-rc.1.22408.1...
Installing pack Microsoft.NET.Runtime.Emscripten.3.1.12.Node.linux-x64 version 7.0.0-rc.1.22405.3...
Workload installation failed. Rolling back installed packs...
Rolling back pack Microsoft.NET.Runtime.Emscripten.3.1.12.Node.linux-x64 installation...
Rolling back pack Microsoft.NET.Runtime.MonoTargets.Sdk installation...
Uninstalling workload pack Microsoft.NET.Runtime.MonoTargets.Sdk version 7.0.0-rc.1.22408.1…
Rolling back pack Microsoft.NET.Runtime.MonoAOTCompiler.Task installation...
Uninstalling workload pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 7.0.0-rc.1.22408.1…
Rolling back pack Microsoft.NETCore.App.Runtime.AOT.linux-x64.Cross.browser-wasm installation...
Uninstalling workload pack Microsoft.NETCore.App.Runtime.AOT.Cross.browser-wasm version 7.0.0-rc.1.22408.1…
Rolling back pack Microsoft.NETCore.App.Runtime.Mono.browser-wasm installation...
Uninstalling workload pack Microsoft.NETCore.App.Runtime.Mono.browser-wasm version 7.0.0-rc.1.22408.1…
Rolling back pack Microsoft.NET.Runtime.WebAssembly.Sdk installation...
Uninstalling workload pack Microsoft.NET.Runtime.WebAssembly.Sdk version 7.0.0-rc.1.22408.1…
Workload installation failed: microsoft.net.runtime.emscripten.3.1.12.node.linux-x64::7.0.0-rc.1.22405.3 is not found in NuGet feeds https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5-transport/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet3.1/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet3.1-transport/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/benchmark-dotnet-prerelease/nuget/v3/index.json".

The nuget.config file defined in dotnet/performance contains a long list of nuget feeds: https://github.com/dotnet/performance/blob/main/NuGet.config . Which one needs to be added?

@radical @naricc @SamMonoRT

radical commented 2 years ago

This is needed - <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />.

  1. Install 7.0 sdk to run benchmarks against
  2. dotnet workload install wasm-tools --skip-manifest-update
  3. Download https://raw.githubusercontent.com/dotnet/runtime/main/src/mono/wasm/test-main.js into /tmp/data/test-main.js
  4. In a checkout for dotnet/performance, run: 
    $ python3 ./scripts/benchmarks_ci.py --csproj src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture x64 -f net7.0 --cli-source-info args --cli-branch refs/heads/main --cli-repository https://github.com/dotnet/runtime --run-isolated --wasm --bdn-artifacts artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoWASM NoMono --buildTimeout 3600 --logBuildOutput --generateBinLog --aotcompilermode wasm --wasmDataDir /tmp/data"

This is what I have been using, and based it on the CI runs.

adamsitnik commented 2 years ago

This is needed - .

It's already in the nuget.config: https://github.com/dotnet/performance/blob/3e63b12c2cfe8d3f4f8b6083e49ab28b5111cedd/NuGet.config#L9

adamsitnik commented 2 years ago

@radical thanks for sharing your steps!

I got to the point where I was getting:

[2022/09/10 13:25:05][INFO]             .withExitOnUnhandledError()
[2022/09/10 13:25:05][INFO]              ^
[2022/09/10 13:25:05][INFO] TypeError: dotnet.withVirtualWorkingDirectory(...).withEnvironmentVariables(...).withDiagnosticTracing(...).withExitOnUnhandledError is not a function
[2022/09/10 13:25:05][INFO]     at run (test-main.js:262:14)
[2022/09/10 13:25:05][INFO] 
[2022/09/10 13:25:05][INFO] 1 pending unhandled Promise rejection(s) detected.
[2022/09/10 13:25:05][INFO] No Workload Results were obtained from the run.

but I just commented out this line of test-main.js and it works now.

FWIW the command that I've used:

python3 ./scripts/benchmarks_ci.py --filter '*Burgers.Test0' -f net7.0 --bdn-arguments "--runtimes wasmnet7.0 --wasmDataDir /home/adam/projects/repros/wasm/"