search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.43k stars 960 forks source link

Detect when MonoAOT compilation fails and abort the benchmark run #2311

Closed caaavik-msft closed 1 year ago

caaavik-msft commented 1 year ago

When BDN builds a MonoAOT project, it first does a regular dotnet build, then it does the MonoAOT compilation (source). These two builds both output to the same directory, and so there is a dependency that the MonoAOT compilation will overwrite the DLLs produced during the original dotnet build.

What we are seeing in the .NET perf lab is that a change caused the MonoAOT compilation to fail for our benchmarks, but BDN did not abort the benchmark run, and instead continued to run the benchmarks against the output from the original dotnet build, meaning it was instead measuring CoreCLR. There should be a check added to ensure that if the MonoAOT compilation fails that we do not continue running the benchmarks and return an error.

I don't have a good repro at the moment for this to debug it further, but will try get some more information soon after speaking with the Mono team.

adamsitnik commented 1 year ago

cc @naricc @radical

radical commented 1 year ago

A log of the failure would be useful. We do get aot compilation failures on wasm perf runs sometimes, and the build does stop after that.

radical commented 1 year ago

We had a similar issue in wasm and had fixed that with - https://github.com/dotnet/benchmarkdotnet/commit/ca43aafc6dbde588ef059e48150100364e4a60f2 .

radical commented 1 year ago

Though in this case, looking at an older log with aot failures, the --no-dependencies build also fails, and yet the benchmarks are run.

[2023/06/05 06:50:32][INFO] /home/helixbot/work/A5DA08AA/w/B80309FE/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/9e6f190d-5e70-4c05-b7e4-c543f1136636/BenchmarkDotNet.Autogenerated.csproj(55,4): error : Precompiling failed for /home/helixbot/work/A5DA08AA/w/B80309FE/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/9e6f190d-5e70-4c05-b7e4-c543f1136636/bin/net8.0/linux-x64/publish/Reporting.dll
[2023/06/05 06:50:32][INFO]     0 Warning(s)
[2023/06/05 06:50:32][INFO]     12 Error(s)
[2023/06/05 06:50:32][INFO] Time Elapsed 00:00:01.93
[2023/06/05 06:50:32][INFO] // command took 2.14s and exited with 1
[2023/06/05 06:50:32][INFO] // ***** Done, took 00:00:10 (10.34 sec)   *****
[2023/06/05 06:50:32][INFO] // Found 1 benchmarks:
[2023/06/05 06:50:32][INFO] //   ByteMark.BenchAssignRectangular: Job-LNAJOD(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

@adamsitnik Something is ignoring the failure here? I'm not familiar enough to know the codepath being followed.

Edit: this was from https://dev.azure.com/dnceng/internal/_build/results?buildId=2193810&view=results

radical commented 1 year ago

@LoopedBard3 ^

kotlarmilos commented 1 year ago

When BDN builds a MonoAOT project, it first does a regular dotnet build, then it does the MonoAOT compilation (source). These two builds both output to the same directory, and so there is a dependency that the MonoAOT compilation will overwrite the DLLs produced during the original dotnet build.

One option may be to dump the output directory before runs, which may increase the build time.

Though in this case, looking at an older log with aot failures, the --no-dependencies build also fails, and yet the benchmarks are run.

Sounds like a good approach when there is an issue with dependencies.

Generally, is there a notification system or watchdog in place to inform us if microbenchmarks haven't been executed at the desired frequency?

kotlarmilos commented 1 year ago

@caaavik-msft Can you provide a command used on the CI for building the runtime using the MonoAotBuilder.cs? We want to reproduce it locally in order to patch it.

radical commented 1 year ago

I traced the code, and found: https://github.com/dotnet/BenchmarkDotNet/blob/58ee5c7a829b225a4211c0ea763ac37104acc021/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs#L58-L81

The result from line 78 is passed to ToBuildResult: https://github.com/dotnet/BenchmarkDotNet/blob/58ee5c7a829b225a4211c0ea763ac37104acc021/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommandResult.cs#L37-L41

.. which in this case ignores the failure, and thus the benchmarks continue to run.

Adding the following after line 78, mitigates the issue:

             if (!buildResult.IsSuccess)
                 return BuildResult.Failure(GenerateResult, buildResult.AllInformation);

As I had mentioned before, we had the same issue with wasm and I had fixed that with https://github.com/dotnet/BenchmarkDotNet/commit/ca43aafc6dbde588ef059e48150100364e4a60f2 .

I'm not sure if the above sounds acceptable, because it does change the behavior. Or if some other way might be more idiomatic. I'm not very familiar with the code.

HTH! cc @caaavik-msft @kotlarmilos @adamsitnik @LoopedBard3 @SamMonoRT @lewing

caaavik-msft commented 1 year ago

@caaavik-msft Can you provide a command used on the CI for building the runtime using the MonoAotBuilder.cs? We want to reproduce it locally in order to patch it.

Apologies, didn't see this until now. In this particular pipeline run, the version of dotnet that BDN uses to run the MonoAOT compiler is the one that gets installed using the dotnet/installer: dotnet-install.sh -Architecture arm64 -Channel 8.0 -Quality daily.

The issue itself happened when the SDK version from the installer changed from 8.0.100-preview.5.23260.3 to 8.0.100-preview.5.23261.2. Larry Ewing was able to track down that the root cause of the compilation error was this commit. Not sure if this answers your question or not, but happy to jump in a call or chat to debug this further if you need more information.

SamMonoRT commented 1 year ago

@radical @adamsitnik @caaavik-msft

In my opinion, we should make the change @radical proposes above to ensure we stop on failures and not get measurements for CoreCLR on Mono lanes

DrewScoggins commented 1 year ago

@lewing @radical @SamMonoRT

Here is a stack trace of the actual compilation error in Mono

[2023/07/05 22:23:32][INFO]   [Reporting.dll] Exec (with response file contents expanded) in /home/helixbot/work/A64E09A5/w/C1830A59/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/6b9c34c4-4546-4c56-ad58-acb0db0b30fe/bin/net8.0/linux-x64/publish: MONO_PATH=/home/helixbot/work/A64E09A5/w/C1830A59/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/6b9c34c4-4546-4c56-ad58-acb0db0b30fe/bin/net8.0/linux-x64/publish: MONO_ENV_OPTIONS= /home/helixbot/work/A64E09A5/p/monoaot/sgen/mini/mono-sgen --debug --llvm "--aot=mcpu=native,nodebug,llvm-path=/home/helixbot/work/A64E09A5/p/monoaot/pack/runtimes/linux-x64/native,outfile=/home/helixbot/work/A64E09A5/w/C1830A59/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/6b9c34c4-4546-4c56-ad58-acb0db0b30fe/bin/net8.0/linux-x64/publish/Reporting.dll.so,llvm-outfile=/home/helixbot/work/A64E09A5/w/C1830A59/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/6b9c34c4-4546-4c56-ad58-acb0db0b30fe/bin/net8.0/linux-x64/publish/Reporting.dll-llvm.o" "Reporting.dll"
[2023/07/05 22:23:32][INFO] 
[2023/07/05 22:23:32][INFO]   Mono Warning: --llvm not enabled in this runtime.
[2023/07/05 22:23:32][INFO]   * Assertion at /__w/1/s/src/mono/mono/metadata/assembly.c:2718, condition `corlib' not met
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]     Native Crash Reporting
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]   Got a SIGABRT while executing native code. This usually indicates
[2023/07/05 22:23:32][INFO]   a fatal error in the mono runtime or one of the native libraries
[2023/07/05 22:23:32][INFO]   used by your application.
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]     Native stacktrace:
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]     0x5567d1630921 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d15d887e - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d16301c1 - Unknown
[2023/07/05 22:23:32][INFO]     0x7f5a7f713980 - Unknown
[2023/07/05 22:23:32][INFO]     0x7f5a7e977e87 - Unknown
[2023/07/05 22:23:32][INFO]     0x7f5a7e9797f1 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d14065a5 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d140f656 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d1406a0f - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d1406b75 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d1406bb7 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d146d5fa - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d14680e6 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d1537d91 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d159b58b - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d13a3eda - Unknown
[2023/07/05 22:23:32][INFO]     0x7f5a7e95ac87 - Unknown
[2023/07/05 22:23:32][INFO]     0x5567d13a3da9 - Unknown
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]     External Debugger Dump:
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]   [New LWP 723]
[2023/07/05 22:23:32][INFO]   Dwarf Error: Cannot handle DW_FORM_<unknown> in DWARF reader [in module /home/helixbot/work/A64E09A5/p/monoaot/sgen/mini/mono-sgen.dbg]
[2023/07/05 22:23:32][INFO]   [Thread debugging using libthread_db enabled]
[2023/07/05 22:23:32][INFO]   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[2023/07/05 22:23:32][INFO]   0x00007f5a7f71332a in __waitpid (pid=732, stat_loc=0x7ffe09f2a800, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
[2023/07/05 22:23:32][INFO]   30    ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
[2023/07/05 22:23:32][INFO]     Id   Target Id         Frame
[2023/07/05 22:23:32][INFO]   * 1    Thread 0x7f5a7fb29740 (LWP 716) "mono-sgen" 0x00007f5a7f71332a in __waitpid (pid=732, stat_loc=0x7ffe09f2a800, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
[2023/07/05 22:23:32][INFO]     2    Thread 0x7f5a7dfff700 (LWP 723) "SGen worker" 0x00007f5a7f70ead3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5567d16883a8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[2023/07/05 22:23:32][INFO]   Thread 2 (Thread 0x7f5a7dfff700 (LWP 723)):
[2023/07/05 22:23:32][INFO]   #0  0x00007f5a7f70ead3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5567d16883a8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[2023/07/05 22:23:32][INFO]   #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x5567d1688358 <lock>, cond=0x5567d1688380 <work_cond>) at pthread_cond_wait.c:502
[2023/07/05 22:23:32][INFO]   #2  __pthread_cond_wait (cond=0x5567d1688380 <work_cond>, mutex=0x5567d1688358 <lock>) at pthread_cond_wait.c:655
[2023/07/05 22:23:32][INFO]   #3  0x00005567d1463723 in thread_func ()
[2023/07/05 22:23:32][INFO]   #4  0x00007f5a7f7086db in start_thread (arg=0x7f5a7dfff700) at pthread_create.c:463
[2023/07/05 22:23:32][INFO]   #5  0x00007f5a7ea5a61f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
[2023/07/05 22:23:32][INFO]   Thread 1 (Thread 0x7f5a7fb29740 (LWP 716)):
[2023/07/05 22:23:32][INFO]   #0  0x00007f5a7f71332a in __waitpid (pid=732, stat_loc=0x7ffe09f2a800, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
[2023/07/05 22:23:32][INFO]   #1  0x00005567d1630a52 in mono_dump_native_crash_info ()
[2023/07/05 22:23:32][INFO]   #2  0x00005567d15d887e in mono_handle_native_crash ()
[2023/07/05 22:23:32][INFO]   #3  0x00005567d16301c1 in sigabrt_signal_handler ()
[2023/07/05 22:23:32][INFO]   #4  <signal handler called>
[2023/07/05 22:23:32][INFO]   #5  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
[2023/07/05 22:23:32][INFO]   #6  0x00007f5a7e9797f1 in __GI_abort () at abort.c:79
[2023/07/05 22:23:32][INFO]   #7  0x00005567d14065a5 in monoeg_assert_abort ()
[2023/07/05 22:23:32][INFO]   #8  0x00005567d140f656 in mono_log_write_logfile ()
[2023/07/05 22:23:32][INFO]   #9  0x00005567d1406a0f in monoeg_g_logv_nofree ()
[2023/07/05 22:23:32][INFO]   #10 0x00005567d1406b75 in monoeg_assertion_message ()
[2023/07/05 22:23:32][INFO]   #11 0x00005567d1406bb7 in mono_assertion_message ()
[2023/07/05 22:23:32][INFO]   #12 0x00005567d146d5fa in mono_assembly_load_corlib ()
[2023/07/05 22:23:32][INFO]   #13 0x00005567d14680e6 in mono_init ()
[2023/07/05 22:23:32][INFO]   #14 0x00005567d1537d91 in mini_init ()
[2023/07/05 22:23:32][INFO]   #15 0x00005567d159b58b in mono_main ()
[2023/07/05 22:23:32][INFO]   #16 0x00005567d13a3eda in main ()
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]     Basic Fault Address Reporting
[2023/07/05 22:23:32][INFO]   =================================================================
[2023/07/05 22:23:32][INFO]   Memory around native instruction pointer (0x7f5a7e977e87):0x7f5a7e977e77  d2 4c 89 ce bf 02 00 00 00 b8 0e 00 00 00 0f 05  .L..............
[2023/07/05 22:23:32][INFO]   0x7f5a7e977e87  48 8b 8c 24 08 01 00 00 64 48 33 0c 25 28 00 00  H..$....dH3.%(..
[2023/07/05 22:23:32][INFO]   0x7f5a7e977e97  00 44 89 c0 75 1f 48 81 c4 18 01 00 00 c3 0f 1f  .D..u.H.........
[2023/07/05 22:23:32][INFO]   0x7f5a7e977ea7  00 48 8b 15 b9 bf 3a 00 f7 d8 41 b8 ff ff ff ff  .H....:...A.....
adamsitnik commented 1 year ago

In my opinion, we should make the change @radical proposes above to ensure we stop on failures and not get measurements for CoreCLR on Mono lanes

I agree.

SamMonoRT commented 1 year ago

From @caaavik-msft : [Yesterday 5:50 PM] Cameron Aavik

it seems to still be failing, but there is a different error now https://helix.dot.net/api/jobs/c4424d7f-c446-4316-8e14-40d270957040/workitems/arm64.micro_mono.net8.0.Partition0?api-version=2019-06-17&access_token=accessToken

put your helix access token in and look at the ConsoleOutputUri field

looks like the mono-sgen is not being placed in the expected directory, I will talk to Parker and see what is happening

oh actually mono-sgen is there, it just can't find libc++.so.1

/home/helixbot/work/A13A0852/p/monoaot/sgen/mini/mono-sgen: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

SamMonoRT commented 1 year ago

@DrewScoggins can you invite "@directhex" to this repo. He might have suggestions on the libc++.so error. @radical while he investigates the mono aot compilation error as well.

directhex commented 1 year ago

How, precisely, does mono-sgen migrate to the /home/helixbot/work/A13A0852/p/monoaot/sgen/mini/ folder? What is responsible for that file existing in that folder?

DrewScoggins commented 1 year ago

I don't think I have "invite" power :), but it seems like they are here.

caaavik-msft commented 1 year ago

From @caaavik-msft : [Yesterday 5:50 PM] Cameron Aavik

it seems to still be failing, but there is a different error now https://helix.dot.net/api/jobs/c4424d7f-c446-4316-8e14-40d270957040/workitems/arm64.micro_mono.net8.0.Partition0?api-version=2019-06-17&access_token=accessToken

put your helix access token in and look at the ConsoleOutputUri field

looks like the mono-sgen is not being placed in the expected directory, I will talk to Parker and see what is happening

oh actually mono-sgen is there, it just can't find libc++.so.1

/home/helixbot/work/A13A0852/p/monoaot/sgen/mini/mono-sgen: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

I have created an issue in the performance repo about this so it can be tracked separately https://github.com/dotnet/performance/issues/3112

kotlarmilos commented 1 year ago

The job dotnet-runtime-perf -> Performance linux x64 release mono AOT micro_mono perftiger running on Ubuntu.1804.Amd64.Tiger.Perf is failing with 

Assertion at /__w/1/s/src/mono/mono/metadata/assembly.c:2718, condition `corlib' not met

which is a Mono-specific issue.

The job dotnet-runtime-perf-slow -> Performance linux arm64 release mono AOT micro_mono perfampere running on Ubuntu.2004.Arm64.Perf is failing with 

Assertion at /__w/1/s/src/mono/mono/metadata/assembly.c:2718, condition `corlib' not met

and

error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

which is a platform-specific issue also present in coreclr runs.

To fix the Mono AOT-specific issue, I added --self-contained parameter to the microbenchmarks build in https://dev.azure.com/dnceng/internal/_git/dotnet-runtime?path=%2F&version=GBbugfix%2Fself-contained-microbenchmarks&_a=history but it seems that the runtime/performance needs to be updated. @cincuranet since you worked on the related issue in https://github.com/dotnet/performance/pull/2730, you may be able to help us further (/cc: @LoopedBard3).

The libc++.so issue may be caused by a corrupted package path or missing global installation.

How, precisely, does mono-sgen migrate to the /home/helixbot/work/A13A0852/p/monoaot/sgen/mini/ folder? What is responsible for that file existing in that folder?

In the eng/testing/performance/performance-setup.sh you'll find the monoaot_path variable that specifies the payload. https://github.com/dotnet/runtime/blob/a2a1885f8d0f66546788c758c62013c3da10de86/eng/testing/performance/performance-setup.sh#L438-L442

During the "Create aot directory (Linux)" step, artifacts are copied from the build directory to the monoaot_path directory. https://github.com/dotnet/runtime/blob/a2a1885f8d0f66546788c758c62013c3da10de86/eng/pipelines/coreclr/templates/perf-job.yml#L213-L214

Once the issues are resolved, we should ensure they don't happen again. @radical, your changes in https://github.com/dotnet/BenchmarkDotNet/issues/2311#issuecomment-1613903639 look good to me and I suggest opening a PR to retrieve feedback from the area owners. Also, I suggest we create a mailing list and subscribe to all failing perf jobs to act timely and ensure the green CI.

directhex commented 1 year ago

Great, thanks @kotlarmilos, that's the info I needed.

So libc++ is a new dependency for Mono due to LLVM 16 requiring C++17 (which is unsupported by libstdc++ on Mariner). We're using rpath to tell Mono that it should look in the same directory ($ORIGIN), and copying it to the right place in the bin/ folder via msbuild logic in mono.proj. Neither of these are accounted for when just grabbing mid-build output from obj/, as is done by the performance build. Either the performance job steps need to account for the changes (setting rpath & copying libc++, or copying from bin/ which is already correct) or we need to alter the build overall to ensure that by the time mono-sgen is compiled in obj/, it already has the right files in the right places.

directhex commented 1 year ago

We're using CMake's install-time rpath logic, CMAKE_INSTALL_RPATH, so someone would need to find a way to switch that to build-time logic instead if we want Mono files in obj/ to have the correct value (as-is the rpath value is set in the install target when files transition from obj/ to bin/). Plus we need libc++.so.1 and libc++abi.so.1 to get into the build tree "properly" (I did it in 30 minutes with MSBuild after wasting a day trying to make CMake do it)

kotlarmilos commented 1 year ago

Repro command for the Mono failure on osx-arm64.

Build the runtime:

./build.sh mono+libs+clr.hosts -c Release /p:MonoEnableLlvm=true /p:MonoLLVMUseCxx11Abi=true

Run the microbenchmark CI script:

python3 dotnet/performance/scripts/benchmarks_ci.py --csproj dotnet/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture arm64 -f net8.0 --upload-to-perflab-container --dotnet-versions 8.0.100-preview.7.23359.1 --cli-source-info args --cli-branch refs/heads/main --cli-commit-sha caff01c55e92d98b8ec077efc22783d24a69b198 --cli-repository https://github.com/dotnet/runtime --cli-source-timestamp 2023-07-10T03:14:43Z --bdn-artifacts dotnet/runtime-test/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath dotnet/runtime-test/artifacts/obj/mono/OSX.arm64.Release/out/bin/mono-sgen --customruntimepack dotnet/runtime-test/artifacts/bin/microsoft.netcore.app.runtime.osx-arm64/Release --aotcompilermode llvm --logBuildOutput --generateBinLog  --partition-count 30 --partition-index 25"
directhex commented 1 year ago

@kotlarmilos I can't find any links to Helix logs which are still available (as opposed to expired/gone). Is the libc++ issue happening only on Ubuntu 20.04 ARM64?

kotlarmilos commented 1 year ago

Correct, it is happening only on Ubuntu arm64. Here is a build with the issue: https://dev.azure.com/dnceng/internal/_build/results?buildId=2217870&view=logs&j=8e5e942d-69f3-5310-4122-1f08fc5161c2

Make sure to pass the access_token, otherwise it has unscheduled status.

directhex commented 1 year ago

Proposed fix in https://github.com/dotnet/runtime/pull/88705

radical commented 1 year ago

With bdn PR https://github.com/dotnet/BenchmarkDotNet/pull/2367, and https://github.com/dotnet/performance/pull/3122 , I can AOT.

  [221/222] Microsoft.DotNet.PlatformAbstractions.dll -> Microsoft.DotNet.PlatformAbstractions.dll.dylib, Microsoft.DotNet.PlatformAbstractions.dll-llvm.o
  [222/222] Microsoft.Extensions.Caching.Abstractions.dll -> Microsoft.Extensions.Caching.Abstractions.dll.dylib, Microsoft.Extensions.Caching.Abstractions.dll-llvm.o
  CompiledAssemblies:
Build succeeded.
CSC : warning CS8002: Referenced assembly 'MicroBenchmarks, Version=42.42.42.42, Culture=neutral, PublicKeyToken=null' does not have a strong name. [/Users/ankj/dev/performance/artifacts/bin/for-running/MicroBenchmarks/Job-JPBQUG/BenchmarkDotNet.Autogenerated.csproj]
    1 Warning(s)
    0 Error(s)
Time Elapsed 00:03:42.81
// command took 223.06 sec and exited with 0
// ***** Done, took 00:03:49 (229.01 sec)   *****
// Found 1 benchmarks:
//   LinqBenchmarks.Where01LinqQueryX: Job-JPBQUG(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
radical commented 1 year ago

The BDN PR also causes the whole run to stop on build failures:

...
[2023/07/11 21:21:17][INFO] /Users/ankj/dev/performance/artifacts/bin/for-running/MicroBenchmarks/Job-TLZJEB/BenchmarkDotNet.Autogenerated.csproj(99,4): error : Precompiling failed for /Users/ankj/dev/performance/artifacts/bin/for-
running/MicroBenchmarks/Job-TLZJEB/bin/net8.0/osx-arm64/publish/Microsoft.Extensions.Caching.Abstractions.dll
[2023/07/11 21:21:17][INFO]     0 Warning(s)
[2023/07/11 21:21:17][INFO]     10 Error(s)
[2023/07/11 21:21:17][INFO] Time Elapsed 00:00:01.39
[2023/07/11 21:21:17][INFO]
[2023/07/11 21:21:17][INFO] // BenchmarkDotNet has failed to build the auto-generated boilerplate code.
...
adamsitnik commented 1 year ago

Is my understanding correct that #2367 has fixed this issue?

radical commented 1 year ago

Is my understanding correct that https://github.com/dotnet/BenchmarkDotNet/pull/2367 has fixed this issue?

@kotlarmilos can confirm that.

kotlarmilos commented 1 year ago

I confirm that the issue has been fixed. Thanks! @caaavik-msft Feel free to close this issue.

timcassell commented 1 year ago

Thanks for the confirmation @kotlarmilos.