search

dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
700 stars 273 forks source link

Investigate and Reenable Perf_GC AllocateArray tests for wasm #2103

Open LoopedBard3 opened 3 years ago

LoopedBard3 commented 3 years ago

Tracking reversal of changes made by #2102. The Allocate Array tests should have the NoWasm category removed once the tests will succeed.

The root cause of the failure is unknown. Testing different argument passing techniques did not change the failure. Both the AllocateArray and AllocateUninitializedArray benchmarks cause failures.

Log Error:

[2021/10/26 09:29:32][INFO] // Found 1 benchmarks:
[2021/10/26 09:29:32][INFO] //   Perf_GC<Char>.AllocateArray: Job-WWKWXV(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=Wasm, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [length=1000, pinned=True]
[2021/10/26 09:29:32][INFO] 
[2021/10/26 09:29:32][INFO] // **************************
[2021/10/26 09:29:32][INFO] // Benchmark: Perf_GC<Char>.AllocateArray: Job-WWKWXV(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=Wasm, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [length=1000, pinned=True]
[2021/10/26 09:29:32][INFO] // *** Execute ***
[2021/10/26 09:29:32][INFO] // Launch: 1 / 1
[2021/10/26 09:29:32][INFO] // Execute: /home/helixbot/.jsvu/v8 --expose_wasm runtime.js -- --run bd7b6a46-442c-47a5-a09c-8c9a85e38fa3.dll --benchmarkName "System.Tests.Perf_GC<Char>.AllocateArray(length: 1000, pinned: True)" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=Wasm, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1" --benchmarkId 77  in /home/helixbot/work/B8460A0E/p/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/bd7b6a46-442c-47a5-a09c-8c9a85e38fa3/bin/net7.0/browser-wasm/AppBundle
[2021/10/26 09:29:32][INFO] Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
[2021/10/26 09:29:33][INFO] console.error: Error: Garbage collector could not allocate 65536u bytes of memory for major heap section.
[2021/10/26 09:29:33][INFO] console.info: Arguments: --run,bd7b6a46-442c-47a5-a09c-8c9a85e38fa3.dll,--benchmarkName,System.Tests.Perf_GC<Char>.AllocateArray(length: 1000, pinned: True),--job,PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=Wasm, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1,--benchmarkId,77
[2021/10/26 09:29:33][INFO] console.debug: MONO_WASM: Initializing mono runtime
[2021/10/26 09:29:33][INFO] console.debug: MONO_WASM: ICU data archive(s) loaded, disabling invariant mode
[2021/10/26 09:29:33][INFO] console.debug: mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
[2021/10/26 09:29:33][INFO] console.info: Initializing.....
[2021/10/26 09:29:33][INFO] // BeforeAnythingElse
[2021/10/26 09:29:33][INFO] 
[2021/10/26 09:29:33][INFO] // Benchmark Process Environment Information:
[2021/10/26 09:29:33][INFO] // Runtime=.NET Core (Mono) 7.0.0-ci, Wasm AOT
[2021/10/26 09:29:33][INFO] // GC=Non-concurrent Workstation
[2021/10/26 09:29:33][INFO] // Job: Job-PLKVBT(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2021/10/26 09:29:33][INFO] 
[2021/10/26 09:29:33][INFO] OverheadJitting  1: 1 op, 120000.00 ns, 120.0000 us/op
[2021/10/26 09:29:33][INFO] WorkloadJitting  1: 1 op, 72000.00 ns, 72.0000 us/op
[2021/10/26 09:29:33][INFO] 
[2021/10/26 09:29:33][INFO] OverheadJitting  2: 16 op, 236000.00 ns, 14.7500 us/op
[2021/10/26 09:29:33][INFO] WorkloadJitting  2: 16 op, 241000.00 ns, 15.0625 us/op
[2021/10/26 09:29:33][INFO] 
[2021/10/26 09:29:33][INFO] WorkloadPilot    1: 16 op, 6000.00 ns, 375.0000 ns/op
[2021/10/26 09:29:33][INFO] WorkloadPilot    2: 666672 op, 516879000.00 ns, 775.3123 ns/op
[2021/10/26 09:29:33][INFO] WorkloadPilot    3: 322464 op, 69279000.00 ns, 214.8426 ns/op
[2021/10/26 09:29:33][INFO] // Benchmark Process 9916 has exited with code 1.
[2021/10/26 09:29:33][INFO] Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
[2021/10/26 09:29:33][INFO]    at System.Linq.ThrowHelper.ThrowNoMatchException()
[2021/10/26 09:29:33][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
[2021/10/26 09:29:33][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
[2021/10/26 09:29:33][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
[2021/10/26 09:29:33][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
[2021/10/26 09:29:33][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
[2021/10/26 09:29:33][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
[2021/10/26 09:29:33][INFO]    at MicroBenchmarks.Program.Main(String[] args) in /home/helixbot/work/B8460A0E/p/performance/src/benchmarks/micro/Program.cs:line 42
[2021/10/26 09:29:33][INFO] $ popd
[2021/10/26 09:29:33][ERROR] Process exited with status 134
[2021/10/26 09:29:33][INFO] Run failure registered
Traceback (most recent call last):
  File "/home/helixbot/work/B8460A0E/p/performance/scripts/benchmarks_ci.py", line 250, in <module>
    __main(sys.argv[1:])
  File "/home/helixbot/work/B8460A0E/p/performance/scripts/benchmarks_ci.py", line 231, in __main
    args
  File "/home/helixbot/work/B8460A0E/p/performance/scripts/micro_benchmarks.py", line 314, in run
    *run_args
  File "/home/helixbot/work/B8460A0E/p/performance/scripts/dotnet.py", line 468, in run
    self.working_directory)
  File "/home/helixbot/work/B8460A0E/p/performance/scripts/performance/common.py", line 219, in run
    returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ dotnet run --project /home/helixbot/work/B8460A0E/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net7.0 --no-restore --no-build -- --anyCategories Libraries Runtime "" --category-exclusion-filter NoInterpreter NoWASM NoMono --wasmEngine /home/helixbot/.jsvu/v8 --runtimeSrcDir /home/helixbot/work/B8460A0E/p/dotnet-wasm "" --partition-count 30 --partition-index 15 --artifacts /home/helixbot/work/B8460A0E/p/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B8460A0E/p/performance/artifacts/packages --runtimes wasm --buildTimeout 600' returned non-zero exit status 134.
+ export _commandExitCode=1

cc @naricc

LoopedBard3 commented 3 years ago

It may be valuable to check if these is supposed to be supported on Wasm as the test may not be Wasm Compatible in general.