search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.5k stars 965 forks source link

Using `DisassemblyDiagnoser` in GitHub Actions #2404

Closed eduherminio closed 1 year ago

eduherminio commented 1 year ago

I attempted to add DisassemblyDiagnoser to a benchmark class and run it in GH Actions default runners and I haven't been successful with Linux and macOS.

Linux

Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied`

After running it with sudo:

Process completed with exit code 139

macOS

(similar to https://github.com/dotnet/BenchmarkDotNet/issues/1499):

// * Diagnostic Output - DisassemblyDiagnoser *
No benchmarks were disassembled

Interestingly, when ran with sudo an asm-md file was created, but an empty one.

Link to the actual benchmark runs: https://github.com/lynx-chess/Lynx/actions/runs/5882072009 https://github.com/lynx-chess/Lynx/actions/runs/5882730576 (with sudo)

Branch where I tried to add the DisassemblyDiagnoser: https://github.com/lynx-chess/Lynx/tree/dotnet/90667

adamsitnik commented 1 year ago

Hi @eduherminio

139 is SIGSEGV, which .NET and BDN version did you use? It looks like https://github.com/dotnet/BenchmarkDotNet/issues/2070 which was solved in .NET 8 and backported to 7: https://github.com/dotnet/runtime/pull/79846

adamsitnik commented 1 year ago

BTW macOS is currently not supported: https://github.com/dotnet/BenchmarkDotNet/issues/1499#issuecomment-1186998224

eduherminio commented 1 year ago

Hi @eduherminio Eduardo Caceres FTE

139 is SIGSEGV, which .NET and BDN version did you use? It looks like #2070 which was solved in .NET 8 and backported to 7: dotnet/runtime#79846

I faced 139 using .NET SDK 8.0.100-preview.7.23376.3 and BDN 0.13.7 in GH actions default Ubuntu runner.

I just tried locally as well (within WSL) with sdk 7.0.306 and same BDN version, and got the 139 as well

Local logs ```bash $ dotnet run -c Release Available Benchmarks: #0 ArrayCopy #1 BitBoard_Struct_ReadonlyStruct_Class_Record #2 Branching #3 CountBits #4 DivideByHalf #5 EncodeMove #6 EnumCasting #7 FENGeneration #8 GetLS1BIndex #9 GoCommandParsing #10 GoCommandParsing_GeneratedAttribute #11 InitializeBishopAndRookAttacks #12 IsSquareAttacked_local_vs_external #13 IsSquareAttacked_parallel #14 LocalVariableIn_vs_NoIn #15 MakeUnmakeMove_implementation #16 MakeUnmakeMove_integration #17 MoveGeneratorArrayVsDictionary #18 MoveGeneratorParallel #19 MutableReadonlyBitBoard #20 OppositeSide #21 PawnAttacks #22 PieceOffset_Boolean #23 PieceOffset_Side #24 PositionClone #25 PositionIdGeneration #26 PriorityQueue_Dequeue #27 PriorityQueue_EnqueueRange #28 PVTable_SumVsArrayAccess #29 ResetLS1B #30 ResetLS1BvsWithoutLS1B #31 ToggleBits #32 TranspositionTableIndex You should select the target benchmark(s). Please, print a number of a benchmark (e.g. `0`) or a contained benchmark caption (e.g. `ArrayCopy`). If you want to select few, please separate them with space ` ` (e.g. `1 2 3`). You can also provide the class name in console arguments by using --filter. (e.g. `--filter '*ArrayCopy*'`). Enter the asterisk `*` to select all. 32 // Validating benchmarks: // ***** BenchmarkRunner: Start ***** // ***** Found 2 benchmark(s) in total ***** // ***** Building 1 exe(s) in Parallel: Start ***** // start dotnet restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/edu/dev/tmp/Lynx/src/Lynx.Benchmark/bin/Release/net7.0/8562b748-4afb-4e78-af89-3ae3c1ce4253 // command took 2.23 sec and exited with 0 // start dotnet build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/edu/dev/tmp/Lynx/src/Lynx.Benchmark/bin/Release/net7.0/8562b748-4afb-4e78-af89-3ae3c1ce4253 // command took 5.36 sec and exited with 0 // ***** Done, took 00:00:07 (7.72 sec) ***** // Found 2 benchmarks: // TranspositionTableIndex.Modulo: DefaultJob [size=2097152] // TranspositionTableIndex.Modulo: DefaultJob [size=2097153] // ************************** // Benchmark: TranspositionTableIndex.Modulo: DefaultJob [size=2097152] // *** Execute *** // Launch: 1 / 1 // Execute: dotnet 8562b748-4afb-4e78-af89-3ae3c1ce4253.dll --anonymousPipes 122 123 --benchmarkName "Lynx.Benchmark.TranspositionTableIndex.Modulo(size: 2097152)" --job Default --benchmarkId 0 in /home/edu/dev/tmp/Lynx/src/Lynx.Benchmark/bin/Release/net7.0/8562b748-4afb-4e78-af89-3ae3c1ce4253/bin/Release/net7.0 // BeforeAnythingElse // Benchmark Process Environment Information: // BenchmarkDotNet v0.13.7 // Runtime=.NET 7.0.9 (7.0.923.32018), X64 RyuJIT AVX2 // GC=Concurrent Workstation // HardwareIntrinsics=AVX2,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256 // Job: DefaultJob OverheadJitting 1: 1 op, 588800.00 ns, 588.8000 us/op WorkloadJitting 1: 1 op, 721400.00 ns, 721.4000 us/op OverheadJitting 2: 16 op, 1038300.00 ns, 64.8937 us/op WorkloadJitting 2: 16 op, 2404300.00 ns, 150.2688 us/op WorkloadPilot 1: 16 op, 1554500.00 ns, 97.1562 us/op WorkloadPilot 2: 32 op, 3867500.00 ns, 120.8594 us/op WorkloadPilot 3: 64 op, 6136800.00 ns, 95.8875 us/op WorkloadPilot 4: 128 op, 18774600.00 ns, 146.6766 us/op WorkloadPilot 5: 256 op, 27941800.00 ns, 109.1477 us/op WorkloadPilot 6: 512 op, 53809700.00 ns, 105.0971 us/op WorkloadPilot 7: 1024 op, 103322100.00 ns, 100.9005 us/op WorkloadPilot 8: 2048 op, 226087300.00 ns, 110.3942 us/op WorkloadPilot 9: 4096 op, 403528500.00 ns, 98.5177 us/op WorkloadPilot 10: 8192 op, 786790600.00 ns, 96.0438 us/op OverheadWarmup 1: 8192 op, 34700.00 ns, 4.2358 ns/op OverheadWarmup 2: 8192 op, 45700.00 ns, 5.5786 ns/op OverheadWarmup 3: 8192 op, 46600.00 ns, 5.6885 ns/op OverheadWarmup 4: 8192 op, 44900.00 ns, 5.4810 ns/op OverheadWarmup 5: 8192 op, 41600.00 ns, 5.0781 ns/op OverheadWarmup 6: 8192 op, 40400.00 ns, 4.9316 ns/op OverheadWarmup 7: 8192 op, 40600.00 ns, 4.9561 ns/op OverheadWarmup 8: 8192 op, 45100.00 ns, 5.5054 ns/op OverheadWarmup 9: 8192 op, 77100.00 ns, 9.4116 ns/op OverheadWarmup 10: 8192 op, 46500.00 ns, 5.6763 ns/op OverheadActual 1: 8192 op, 46200.00 ns, 5.6396 ns/op OverheadActual 2: 8192 op, 40400.00 ns, 4.9316 ns/op OverheadActual 3: 8192 op, 32200.00 ns, 3.9307 ns/op OverheadActual 4: 8192 op, 41000.00 ns, 5.0049 ns/op OverheadActual 5: 8192 op, 40700.00 ns, 4.9683 ns/op OverheadActual 6: 8192 op, 45200.00 ns, 5.5176 ns/op OverheadActual 7: 8192 op, 46400.00 ns, 5.6641 ns/op OverheadActual 8: 8192 op, 32200.00 ns, 3.9307 ns/op OverheadActual 9: 8192 op, 31900.00 ns, 3.8940 ns/op OverheadActual 10: 8192 op, 31800.00 ns, 3.8818 ns/op OverheadActual 11: 8192 op, 31900.00 ns, 3.8940 ns/op OverheadActual 12: 8192 op, 32000.00 ns, 3.9062 ns/op OverheadActual 13: 8192 op, 44100.00 ns, 5.3833 ns/op OverheadActual 14: 8192 op, 31900.00 ns, 3.8940 ns/op OverheadActual 15: 8192 op, 34100.00 ns, 4.1626 ns/op OverheadActual 16: 8192 op, 32100.00 ns, 3.9185 ns/op OverheadActual 17: 8192 op, 33600.00 ns, 4.1016 ns/op OverheadActual 18: 8192 op, 41300.00 ns, 5.0415 ns/op OverheadActual 19: 8192 op, 40300.00 ns, 4.9194 ns/op OverheadActual 20: 8192 op, 32200.00 ns, 3.9307 ns/op WorkloadWarmup 1: 8192 op, 789246700.00 ns, 96.3436 us/op WorkloadWarmup 2: 8192 op, 798522000.00 ns, 97.4758 us/op WorkloadWarmup 3: 8192 op, 792974900.00 ns, 96.7987 us/op WorkloadWarmup 4: 8192 op, 787944800.00 ns, 96.1847 us/op WorkloadWarmup 5: 8192 op, 858571800.00 ns, 104.8061 us/op WorkloadWarmup 6: 8192 op, 792109600.00 ns, 96.6931 us/op // BeforeActualRun WorkloadActual 1: 8192 op, 776998500.00 ns, 94.8484 us/op WorkloadActual 2: 8192 op, 790785400.00 ns, 96.5314 us/op WorkloadActual 3: 8192 op, 794547900.00 ns, 96.9907 us/op WorkloadActual 4: 8192 op, 772093200.00 ns, 94.2497 us/op WorkloadActual 5: 8192 op, 796677800.00 ns, 97.2507 us/op WorkloadActual 6: 8192 op, 794390100.00 ns, 96.9714 us/op WorkloadActual 7: 8192 op, 776298400.00 ns, 94.7630 us/op WorkloadActual 8: 8192 op, 799839000.00 ns, 97.6366 us/op WorkloadActual 9: 8192 op, 821026900.00 ns, 100.2230 us/op WorkloadActual 10: 8192 op, 797089600.00 ns, 97.3010 us/op WorkloadActual 11: 8192 op, 777251900.00 ns, 94.8794 us/op WorkloadActual 12: 8192 op, 797092800.00 ns, 97.3014 us/op WorkloadActual 13: 8192 op, 796822300.00 ns, 97.2683 us/op WorkloadActual 14: 8192 op, 794410600.00 ns, 96.9740 us/op WorkloadActual 15: 8192 op, 822079900.00 ns, 100.3516 us/op // AfterActualRun WorkloadResult 1: 8192 op, 776964650.00 ns, 94.8443 us/op WorkloadResult 2: 8192 op, 790751550.00 ns, 96.5273 us/op WorkloadResult 3: 8192 op, 794514050.00 ns, 96.9866 us/op WorkloadResult 4: 8192 op, 772059350.00 ns, 94.2455 us/op WorkloadResult 5: 8192 op, 796643950.00 ns, 97.2466 us/op WorkloadResult 6: 8192 op, 794356250.00 ns, 96.9673 us/op WorkloadResult 7: 8192 op, 776264550.00 ns, 94.7589 us/op WorkloadResult 8: 8192 op, 799805150.00 ns, 97.6325 us/op WorkloadResult 9: 8192 op, 797055750.00 ns, 97.2968 us/op WorkloadResult 10: 8192 op, 777218050.00 ns, 94.8753 us/op WorkloadResult 11: 8192 op, 797058950.00 ns, 97.2972 us/op WorkloadResult 12: 8192 op, 796788450.00 ns, 97.2642 us/op WorkloadResult 13: 8192 op, 794376750.00 ns, 96.9698 us/op // GC: 0 0 0 936 8192 // Threading: 0 0 8192 // AfterAll $ echo $? 139 ```
adamsitnik commented 1 year ago

It's probably https://github.com/dotnet/runtime/issues/90691, let's wait until somebody from the .NET Runtime Team investigates it further.