search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.62k stars 973 forks source link

MonoAOTLLVM run failed to build properly but still ran #2533

Open LoopedBard3 opened 9 months ago

LoopedBard3 commented 9 months ago

We recently found a MonoAOTLLVM performance test run that failed to build but still ran and produced results when we expected it to fail instead. Here is part of the log that demonstrates this:

// start dotnet  build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:NuGetPackageRoot="/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/packages" -bl:Job-NKPLKD-build-no-restore.binlog /p:IntermediateOutputPath="/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/obj/Release/net9.0"\ /p:OutDir="/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish" /p:OutputPath="/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish" --output "/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish" in /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD
MSBuild version 17.10.0-preview-24115-01+b783f61ef for .NET
  Packaged ID: /home/helixbot/work/AE440975/p/monoaot/pack
/home/helixbot/work/AE440975/w/B2600917/e/performance/tools/dotnet/x64/sdk/9.0.100-preview.2.24119.3/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): warning NETSDK1023: A PackageReference for 'Microsoft.NET.Test.Sdk' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/helixbot/work/AE440975/w/B2600917/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net9.0]
  Reporting -> /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish/Reporting.dll
  BenchmarkDotNet.Extensions -> /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish/BenchmarkDotNet.Extensions.dll
  MicroBenchmarks -> /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish/MicroBenchmarks.dll
CSC : warning CS8002: Referenced assembly 'MicroBenchmarks, Version=42.42.42.42, Culture=neutral, PublicKeyToken=null' does not have a strong name. [/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/BenchmarkDotNet.Autogenerated.csproj]
  BenchmarkDotNet.Autogenerated -> /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish/Job-NKPLKD.dll
  BenchmarkDotNet.Autogenerated -> /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/
  [1/240] System.Runtime.Loader.dll -> System.Runtime.Loader.dll.so, System.Runtime.Loader.dll-llvm.o
  [2/240] System.Reflection.dll -> System.Reflection.dll.so, System.Reflection.dll-llvm.o
  [3/240] System.Threading.Timer.dll -> System.Threading.Timer.dll.so, System.Threading.Timer.dll-llvm.o
  [4/240] System.Security.Principal.dll -> System.Security.Principal.dll.so, System.Security.Principal.dll-llvm.o
  [5/240] System.Transactions.dll -> System.Transactions.dll.so, System.Transactions.dll-llvm.o
  [6/240] System.Security.Principal.Windows.dll -> System.Security.Principal.Windows.dll.so, System.Security.Principal.Windows.dll-llvm.o
  [7/240] Microsoft.Extensions.Logging.EventSource.dll -> Microsoft.Extensions.Logging.EventSource.dll.so, Microsoft.Extensions.Logging.EventSource.dll-llvm.o
  [8/240] System.Security.SecureString.dll -> System.Security.SecureString.dll.so, System.Security.SecureString.dll-llvm.o
  [9/240] System.ServiceModel.Web.dll -> System.ServiceModel.Web.dll.so, System.ServiceModel.Web.dll-llvm.o
  [10/240] BenchmarkDotNet.Annotations.dll -> BenchmarkDotNet.Annotations.dll.so, BenchmarkDotNet.Annotations.dll-llvm.o
  [11/240] System.IO.MemoryMappedFiles.dll -> System.IO.MemoryMappedFiles.dll.so, System.IO.MemoryMappedFiles.dll-llvm.o
  [12/240] System.ServiceProcess.dll -> System.ServiceProcess.dll.so, System.ServiceProcess.dll-llvm.o
  [13/240] Microsoft.Extensions.Options.ConfigurationExtensions.dll -> Microsoft.Extensions.Options.ConfigurationExtensions.dll.so, Microsoft.Extensions.Options.ConfigurationExtensions.dll-llvm.o
  [14/240] System.ComponentModel.Primitives.dll -> System.ComponentModel.Primitives.dll.so, System.ComponentModel.Primitives.dll-llvm.o
  [15/240] System.Drawing.Primitives.dll -> System.Drawing.Primitives.dll.so, System.Drawing.Primitives.dll-llvm.o
  [16/240] System.Dynamic.Runtime.dll -> System.Dynamic.Runtime.dll.so, System.Dynamic.Runtime.dll-llvm.o
  [17/240] Microsoft.Extensions.Options.dll -> Microsoft.Extensions.Options.dll.so, Microsoft.Extensions.Options.dll-llvm.o
  [18/240] BenchmarkDotNet.Diagnostics.Windows.dll -> BenchmarkDotNet.Diagnostics.Windows.dll.so, BenchmarkDotNet.Diagnostics.Windows.dll-llvm.o
  [19/240] System.IO.Pipelines.dll -> System.IO.Pipelines.dll.so, System.IO.Pipelines.dll-llvm.o
  [20/240] System.Net.Quic.dll -> System.Net.Quic.dll.so, System.Net.Quic.dll-llvm.o
  [21/240] Microsoft.Extensions.Primitives.dll -> Microsoft.Extensions.Primitives.dll.so, Microsoft.Extensions.Primitives.dll-llvm.o
  [22/240] System.IO.Pipes.AccessControl.dll -> System.IO.Pipes.AccessControl.dll.so, System.IO.Pipes.AccessControl.dll-llvm.o
  [23/240] System.Text.Encoding.CodePages.dll -> System.Text.Encoding.CodePages.dll.so, System.Text.Encoding.CodePages.dll-llvm.o
  [24/240] System.Text.Encoding.dll -> System.Text.Encoding.dll.so, System.Text.Encoding.dll-llvm.o
  [25/240] System.Text.Encoding.Extensions.dll -> System.Text.Encoding.Extensions.dll.so, System.Text.Encoding.Extensions.dll-llvm.o
  [26/240] Microsoft.TestPlatform.AdapterUtilities.dll -> Microsoft.TestPlatform.AdapterUtilities.dll.so, Microsoft.TestPlatform.AdapterUtilities.dll-llvm.o
  [27/240] System.Reflection.Emit.dll -> System.Reflection.Emit.dll.so, System.Reflection.Emit.dll-llvm.o
  [28/240] System.Reflection.Emit.ILGeneration.dll -> System.Reflection.Emit.ILGeneration.dll.so, System.Reflection.Emit.ILGeneration.dll-llvm.o
  [29/240] System.Reflection.Emit.Lightweight.dll -> System.Reflection.Emit.Lightweight.dll.so, System.Reflection.Emit.Lightweight.dll-llvm.o
  [30/240] System.Reflection.Extensions.dll -> System.Reflection.Extensions.dll.so, System.Reflection.Extensions.dll-llvm.o
  [31/240] System.IO.Pipes.dll -> System.IO.Pipes.dll.so, System.IO.Pipes.dll-llvm.o
  [32/240] System.IO.UnmanagedMemoryStream.dll -> System.IO.UnmanagedMemoryStream.dll.so, System.IO.UnmanagedMemoryStream.dll-llvm.o
  [33/240] System.Text.Encodings.Web.dll -> System.Text.Encodings.Web.dll.so, System.Text.Encodings.Web.dll-llvm.o
  [34/240] System.Formats.Asn1.dll -> System.Formats.Asn1.dll.so, System.Formats.Asn1.dll-llvm.o
  [35/240] System.Transactions.Local.dll -> System.Transactions.Local.dll.so, System.Transactions.Local.dll-llvm.o
  [36/240] System.ValueTuple.dll -> System.ValueTuple.dll.so, System.ValueTuple.dll-llvm.o
  [37/240] System.Web.dll -> System.Web.dll.so, System.Web.dll-llvm.o
  [38/240] System.Runtime.Numerics.dll -> System.Runtime.Numerics.dll.so, System.Runtime.Numerics.dll-llvm.o
  [39/240] System.Runtime.Serialization.dll -> System.Runtime.Serialization.dll.so, System.Runtime.Serialization.dll-llvm.o
  [40/240] System.Web.HttpUtility.dll -> System.Web.HttpUtility.dll.so, System.Web.HttpUtility.dll-llvm.o
  [41/240] System.Windows.dll -> System.Windows.dll.so, System.Windows.dll-llvm.o
  [42/240] System.Xml.dll -> System.Xml.dll.so, System.Xml.dll-llvm.o
  [43/240] System.Xml.Linq.dll -> System.Xml.Linq.dll.so, System.Xml.Linq.dll-llvm.o
  [44/240] Microsoft.TestPlatform.CommunicationUtilities.dll -> Microsoft.TestPlatform.CommunicationUtilities.dll.so, Microsoft.TestPlatform.CommunicationUtilities.dll-llvm.o
  [45/240] System.Xml.ReaderWriter.dll -> System.Xml.ReaderWriter.dll.so, System.Xml.ReaderWriter.dll-llvm.o
  [46/240] System.Xml.Serialization.dll -> System.Xml.Serialization.dll.so, System.Xml.Serialization.dll-llvm.o
  [47/240] System.Xml.XDocument.dll -> System.Xml.XDocument.dll.so, System.Xml.XDocument.dll-llvm.o
  [48/240] System.Xml.XmlDocument.dll -> System.Xml.XmlDocument.dll.so, System.Xml.XmlDocument.dll-llvm.o
  [49/240] System.Xml.XmlSerializer.dll -> System.Xml.XmlSerializer.dll.so, System.Xml.XmlSerializer.dll-llvm.o
  [50/240] System.Xml.XPath.dll -> System.Xml.XPath.dll.so, System.Xml.XPath.dll-llvm.o
  [51/240] System.Xml.XPath.XDocument.dll -> System.Xml.XPath.XDocument.dll.so, System.Xml.XPath.XDocument.dll-llvm.o
  [52/240] System.Formats.Cbor.dll -> System.Formats.Cbor.dll.so, System.Formats.Cbor.dll-llvm.o
  [53/240] testhost.dll -> testhost.dll.so, testhost.dll-llvm.o
  [54/240] TraceReloggerLib.dll -> TraceReloggerLib.dll.so, TraceReloggerLib.dll-llvm.o
  [55/240] System.Net.Requests.dll -> System.Net.Requests.dll.so, System.Net.Requests.dll-llvm.o
  [56/240] Microsoft.TestPlatform.CoreUtilities.dll -> Microsoft.TestPlatform.CoreUtilities.dll.so, Microsoft.TestPlatform.CoreUtilities.dll-llvm.o
  [Microsoft.TestPlatform.CrossPlatEngine.dll] Exec (with response file contents expanded) in /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish: MONO_PATH=/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish: MONO_ENV_OPTIONS= /home/helixbot/work/AE440975/p/monoaot/mono-aot-cross --debug --llvm "--aot=mcpu=native,nodebug,llvm-path=/home/helixbot/work/AE440975/p/monoaot/pack/runtimes/linux-x64/native,outfile=/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll.so,llvm-outfile=/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll-llvm.o" "Microsoft.TestPlatform.CrossPlatEngine.dll"

  Unable to compile method 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.IProxyDiscoveryManager Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.TestEngine:GetDiscoveryManager (Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IRequestData,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.DiscoveryCriteria,System.Collections.Generic.IDictionary`2<string, Microsoft.VisualStudio.TestPlatform.ObjectModel.SourceDetail>,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IWarningLogger)' due to: 'VTable setup of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.InProcessProxyDiscoveryManager failed assembly:/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll type:InProcessProxyDiscoveryManager member:(null)'.
  Unable to compile method 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.IProxyExecutionManager Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.TestEngine:GetExecutionManager (Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IRequestData,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.TestRunCriteria,System.Collections.Generic.IDictionary`2<string, Microsoft.VisualStudio.TestPlatform.ObjectModel.SourceDetail>,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IWarningLogger)' due to: 'VTable setup of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.InProcessProxyExecutionManager failed assembly:/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll type:InProcessProxyExecutionManager member:(null)'.
  Unable to compile method 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.IProxyExecutionManager Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.TestEngine:CreateNonParallelExecutionManager (Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IRequestData,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.TestRunCriteria,bool,Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestRuntimeProviderInfo)' due to: 'VTable setup of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager failed assembly:/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll type:ProxyExecutionManager member:(null)'.
  =================================================================
    Native Crash Reporting
  =================================================================
  Got a SIGSEGV while executing native code. This usually indicates
  a fatal error in the mono runtime or one of the native libraries
  used by your application.
  =================================================================
  =================================================================
    Native stacktrace:
  =================================================================
    0x5602bb46c231 - Unknown
    0x5602bb40e4de - Unknown
    0x5602bb36a4e6 - Unknown
    0x7fdbc9042520 - Unknown
    0x5602bb3a12fc - Unknown
    0x5602bb37b5cd - Unknown
    0x5602bb395c4e - Unknown
    0x5602bb375b44 - Unknown
    0x5602bb3ed4cc - Unknown
    0x5602bb3ddb04 - Unknown
    0x5602bb3d940e - Unknown
    0x5602bb3656db - Unknown
    0x5602bb361620 - Unknown
    0x5602bb221bba - Unknown
    0x7fdbc9029d90 - Unknown
    0x7fdbc9029e40 - Unknown
    0x5602bb221a89 - Unknown
  =================================================================
    External Debugger Dump:
  =================================================================
  [New LWP 500043]
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  0x00007fdbc90ea42f in __GI___wait4 (pid=pid@entry=500044, stat_loc=stat_loc@entry=0x7ffdbbfe3a80, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
  30    ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
    Id   Target Id                                           Frame
  * 1    Thread 0x7fdbc92d8480 (LWP 500042) "mono-aot-cross" 0x00007fdbc90ea42f in __GI___wait4 (pid=pid@entry=500044, stat_loc=stat_loc@entry=0x7ffdbbfe3a80, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
    2    Thread 0x7fdbc83ff640 (LWP 500043) "SGen worker"    __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5602bc0dd588 <work_cond+40>) at ./nptl/futex-internal.c:57
  Thread 2 (Thread 0x7fdbc83ff640 (LWP 500043) "SGen worker"):
  #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5602bc0dd588 <work_cond+40>) at ./nptl/futex-internal.c:57
  #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5602bc0dd588 <work_cond+40>) at ./nptl/futex-internal.c:87
  #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5602bc0dd588 <work_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
  #3  0x00007fdbc9093a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5602bc0dd538 <lock>, cond=0x5602bc0dd560 <work_cond>) at ./nptl/pthread_cond_wait.c:503
  #4  ___pthread_cond_wait (cond=0x5602bc0dd560 <work_cond>, mutex=0x5602bc0dd538 <lock>) at ./nptl/pthread_cond_wait.c:627
  #5  0x00005602bb25a233 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/sgen/../../mono/utils/mono-os-mutex.h:219
  #6  get_work (worker_index=0, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:164
  #7  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:195
  #8  0x00007fdbc9094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
  #9  0x00007fdbc9126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  Thread 1 (Thread 0x7fdbc92d8480 (LWP 500042) "mono-aot-cross"):
  #0  0x00007fdbc90ea42f in __GI___wait4 (pid=pid@entry=500044, stat_loc=stat_loc@entry=0x7ffdbbfe3a80, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
  #1  0x00007fdbc90ea3ab in __GI___waitpid (pid=pid@entry=500044, stat_loc=stat_loc@entry=0x7ffdbbfe3a80, options=options@entry=0) at ./posix/waitpid.c:38
  #2  0x00005602bb46c362 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
  #3  mono_dump_native_crash_info (signal=<optimized out>, mctx=mctx@entry=0x7ffdbbfe45f0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
  #4  0x00005602bb40e4de in mono_handle_native_crash (signal=0x5602baefdb48 "SIGSEGV", mctx=0x7ffdbbfe45f0, info=0x7ffdbbfe48b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2967
  #5  0x00005602bb36a4e6 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7ffdbbfe48b0, context=0x7ffdbbfe4780, debug_fault_addr=0x108) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3834
  #6  <signal handler called>
  #7  mono_method_to_ir (cfg=cfg@entry=0x5602bd7965e0, method=method@entry=0x5602bc968190, start_bblock=start_bblock@entry=0x5602bd799b30, end_bblock=end_bblock@entry=0x5602bd799c80, return_var=0x5602bd799ae0, inline_args=<optimized out>, inline_offset=1, is_virtual_call=0) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:12283
  #8  0x00005602bb37b5cd in inline_method (cfg=cfg@entry=0x5602bd7965e0, cmethod=0x5602bc968190, fsig=<optimized out>, fsig@entry=0x5602bd577718, sp=<optimized out>, sp@entry=0x5602bd7999d8, ip=<optimized out>, ip@entry=0x7fdbc923f561 "(-\002", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4852
  #9  0x00005602bb395c4e in mono_method_to_ir (cfg=cfg@entry=0x5602bd7965e0, method=method@entry=0x5602bc968118, start_bblock=0x5602bd799478, start_bblock@entry=0x0, end_bblock=0x5602bd7995c8, end_bblock@entry=0x0, return_var=return_var@entry=0x0, inline_args=inline_args@entry=0x0, inline_offset=0, is_virtual_call=0) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7984
  #10 0x00005602bb375b44 in mini_method_compile (method=<optimized out>, method@entry=0x5602bc968118, opts=<optimized out>, flags=<optimized out>, flags@entry=(JIT_FLAG_AOT | JIT_FLAG_LLVM | JIT_FLAG_USE_CURRENT_CPU), parts=<optimized out>, parts@entry=0, aot_method_index=<optimized out>, aot_method_index@entry=553) at /__w/1/s/src/mono/mono/mini/mini.c:3506
  #11 0x00005602bb3ed4cc in compile_method (acfg=acfg@entry=0x5602bc8d8370, method=0x5602bc968118) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:9518
  #12 0x00005602bb3ddb04 in compile_methods (acfg=0x5602bc8d8370) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:13097
  #13 aot_assembly (ass=<optimized out>, jit_opts=jit_opts@entry=374434303, aot_options=aot_options@entry=0x7ffdbbfe6258) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:15183
  #14 0x00005602bb3d940e in mono_aot_assemblies (assemblies=assemblies@entry=0x5602bc8d6c90, nassemblies=1, jit_opts=374434303, aot_options=<optimized out>) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:15583
  #15 0x00005602bb3656db in main_thread_handler (user_data=user_data@entry=0x7ffdbbfe64e0) at /__w/1/s/src/mono/mono/mini/driver.c:1424
  #16 0x00005602bb361620 in mono_main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:2652
  #17 0x00005602bb221bba in mono_main_with_options (argc=-1130987120, argv=0x8) at /__w/1/s/src/mono/mono/mini/main.c:36
  #18 main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/main.c:88
  [Inferior 1 (process 500042) detached]
  =================================================================
    Basic Fault Address Reporting
  =================================================================
  Memory around native instruction pointer (0x5602bb3a12fc):0x5602bb3a12ec  4c 8b b5 d8 fd ff ff 4c 8b bd 50 ff ff ff 74 2b  L......L..P...t+
  0x5602bb3a12fc  89 90 08 01 00 00 48 8b 40 08 48 39 c8 75 f1 eb  ......H.@.H9.u..
  0x5602bb3a130c  1a 4c 89 e7 e8 fb d1 00 00 48 8b 7d 88 4c 8b b5  .L.......H.}.L..
  0x5602bb3a131c  d8 fd ff ff 4c 8b bd 50 ff ff ff 45 85 ff 79 35  ....L..P...E..y5
/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/BenchmarkDotNet.Autogenerated.csproj(75,4): error : Precompiling failed for /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll
  [57/240] System.Runtime.Serialization.Formatters.dll -> System.Runtime.Serialization.Formatters.dll.so, System.Runtime.Serialization.Formatters.dll-llvm.o
  [58/240] System.Formats.Tar.dll -> System.Formats.Tar.dll.so, System.Formats.Tar.dll-llvm.o
  [59/240] System.ComponentModel.TypeConverter.dll -> System.ComponentModel.TypeConverter.dll.so, System.ComponentModel.TypeConverter.dll-llvm.o
  [60/240] System.Linq.dll -> System.Linq.dll.so, System.Linq.dll-llvm.o
  [61/240] System.Net.Security.dll -> System.Net.Security.dll.so, System.Net.Security.dll-llvm.o
  [62/240] System.Reflection.Metadata.dll -> System.Reflection.Metadata.dll.so, System.Reflection.Metadata.dll-llvm.o
  [63/240] Utf8Json.dll -> Utf8Json.dll.so, Utf8Json.dll-llvm.o
  [64/240] BenchmarkDotNet.Extensions.dll -> BenchmarkDotNet.Extensions.dll.so, BenchmarkDotNet.Extensions.dll-llvm.o
  [65/240] BenchmarkDotNet.TestAdapter.dll -> BenchmarkDotNet.TestAdapter.dll.so, BenchmarkDotNet.TestAdapter.dll-llvm.o
  [66/240] System.Text.Json.dll -> System.Text.Json.dll.so, System.Text.Json.dll-llvm.o
  [67/240] Newtonsoft.Json.dll -> Newtonsoft.Json.dll.so, Newtonsoft.Json.dll-llvm.o
  [68/240] CommandLine.dll -> CommandLine.dll.so, CommandLine.dll-llvm.o
  [69/240] Dia2Lib.dll -> Dia2Lib.dll.so, Dia2Lib.dll-llvm.o
  [70/240] Gee.External.Capstone.dll -> Gee.External.Capstone.dll.so, Gee.External.Capstone.dll-llvm.o
  [71/240] BenchmarkDotNet.dll -> BenchmarkDotNet.dll.so, BenchmarkDotNet.dll-llvm.o
  [72/240] Microsoft.Diagnostics.Runtime.dll -> Microsoft.Diagnostics.Runtime.dll.so, Microsoft.Diagnostics.Runtime.dll-llvm.o
  [73/240] Microsoft.DotNet.PlatformAbstractions.dll -> Microsoft.DotNet.PlatformAbstractions.dll.so, Microsoft.DotNet.PlatformAbstractions.dll-llvm.o
  [74/240] Microsoft.Extensions.Caching.Abstractions.dll -> Microsoft.Extensions.Caching.Abstractions.dll.so, Microsoft.Extensions.Caching.Abstractions.dll-llvm.o
  [75/240] Microsoft.Extensions.Caching.Memory.dll -> Microsoft.Extensions.Caching.Memory.dll.so, Microsoft.Extensions.Caching.Memory.dll-llvm.o
  [76/240] Microsoft.Extensions.Configuration.Abstractions.dll -> Microsoft.Extensions.Configuration.Abstractions.dll.so, Microsoft.Extensions.Configuration.Abstractions.dll-llvm.o
  [77/240] Microsoft.Extensions.Configuration.Binder.dll -> Microsoft.Extensions.Configuration.Binder.dll.so, Microsoft.Extensions.Configuration.Binder.dll-llvm.o
  [78/240] Microsoft.Extensions.Configuration.dll -> Microsoft.Extensions.Configuration.dll.so, Microsoft.Extensions.Configuration.dll-llvm.o
  [79/240] Microsoft.Extensions.Configuration.FileExtensions.dll -> Microsoft.Extensions.Configuration.FileExtensions.dll.so, Microsoft.Extensions.Configuration.FileExtensions.dll-llvm.o
  [80/240] Microsoft.Extensions.Configuration.Json.dll -> Microsoft.Extensions.Configuration.Json.dll.so, Microsoft.Extensions.Configuration.Json.dll-llvm.o
  [81/240] Microsoft.Extensions.Configuration.Xml.dll -> Microsoft.Extensions.Configuration.Xml.dll.so, Microsoft.Extensions.Configuration.Xml.dll-llvm.o
  [82/240] Microsoft.Extensions.DependencyInjection.Abstractions.dll -> Microsoft.Extensions.DependencyInjection.Abstractions.dll.so, Microsoft.Extensions.DependencyInjection.Abstractions.dll-llvm.o
  [83/240] Microsoft.Extensions.DependencyInjection.dll -> Microsoft.Extensions.DependencyInjection.dll.so, Microsoft.Extensions.DependencyInjection.dll-llvm.o
  [84/240] Microsoft.Extensions.Diagnostics.Abstractions.dll -> Microsoft.Extensions.Diagnostics.Abstractions.dll.so, Microsoft.Extensions.Diagnostics.Abstractions.dll-llvm.o
  [85/240] Microsoft.Extensions.Diagnostics.dll -> Microsoft.Extensions.Diagnostics.dll.so, Microsoft.Extensions.Diagnostics.dll-llvm.o
  [86/240] Microsoft.Extensions.FileProviders.Abstractions.dll -> Microsoft.Extensions.FileProviders.Abstractions.dll.so, Microsoft.Extensions.FileProviders.Abstractions.dll-llvm.o
  [87/240] Microsoft.Extensions.FileProviders.Physical.dll -> Microsoft.Extensions.FileProviders.Physical.dll.so, Microsoft.Extensions.FileProviders.Physical.dll-llvm.o
  [88/240] Microsoft.Extensions.FileSystemGlobbing.dll -> Microsoft.Extensions.FileSystemGlobbing.dll.so, Microsoft.Extensions.FileSystemGlobbing.dll-llvm.o
  [89/240] Microsoft.Extensions.Http.dll -> Microsoft.Extensions.Http.dll.so, Microsoft.Extensions.Http.dll-llvm.o
  [90/240] Microsoft.Extensions.Logging.Abstractions.dll -> Microsoft.Extensions.Logging.Abstractions.dll.so, Microsoft.Extensions.Logging.Abstractions.dll-llvm.o
  [91/240] Microsoft.Extensions.Logging.dll -> Microsoft.Extensions.Logging.dll.so, Microsoft.Extensions.Logging.dll-llvm.o
  [92/240] Iced.dll -> Iced.dll.so, Iced.dll-llvm.o
  [93/240] Jil.dll -> Jil.dll.so, Jil.dll-llvm.o
  [94/240] Microsoft.Diagnostics.Tracing.TraceEvent.dll -> Microsoft.Diagnostics.Tracing.TraceEvent.dll.so, Microsoft.Diagnostics.Tracing.TraceEvent.dll-llvm.o
  [95/240] Job-NKPLKD.dll -> Job-NKPLKD.dll.so, Job-NKPLKD.dll-llvm.o
  [96/240] MessagePack.dll -> MessagePack.dll.so, MessagePack.dll-llvm.o
  [97/240] MicroBenchmarks.dll -> MicroBenchmarks.dll.so, MicroBenchmarks.dll-llvm.o
  [98/240] Microsoft.Bcl.AsyncInterfaces.dll -> Microsoft.Bcl.AsyncInterfaces.dll.so, Microsoft.Bcl.AsyncInterfaces.dll-llvm.o
  [99/240] Microsoft.CodeAnalysis.CSharp.dll -> Microsoft.CodeAnalysis.CSharp.dll.so, Microsoft.CodeAnalysis.CSharp.dll-llvm.o
  [100/240] Microsoft.CodeAnalysis.dll -> Microsoft.CodeAnalysis.dll.so, Microsoft.CodeAnalysis.dll-llvm.o
  [101/240] Microsoft.CSharp.dll -> Microsoft.CSharp.dll.so, Microsoft.CSharp.dll-llvm.o
  [102/240] Microsoft.Diagnostics.FastSerialization.dll -> Microsoft.Diagnostics.FastSerialization.dll.so, Microsoft.Diagnostics.FastSerialization.dll-llvm.o
  [103/240] Microsoft.Diagnostics.NETCore.Client.dll -> Microsoft.Diagnostics.NETCore.Client.dll.so, Microsoft.Diagnostics.NETCore.Client.dll-llvm.o
Build FAILED.
/home/helixbot/work/AE440975/w/B2600917/e/performance/tools/dotnet/x64/sdk/9.0.100-preview.2.24119.3/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): warning NETSDK1023: A PackageReference for 'Microsoft.NET.Test.Sdk' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/helixbot/work/AE440975/w/B2600917/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net9.0]
CSC : warning CS8002: Referenced assembly 'MicroBenchmarks, Version=42.42.42.42, Culture=neutral, PublicKeyToken=null' does not have a strong name. [/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/BenchmarkDotNet.Autogenerated.csproj]
/home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/BenchmarkDotNet.Autogenerated.csproj(75,4): error : Precompiling failed for /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publishpublish/Microsoft.TestPlatform.CrossPlatEngine.dll
    2 Warning(s)
    1 Error(s)
Time Elapsed 00:05:11.95
// command took 312.2 sec and exited with 1
// ***** Done, took 00:05:18 (318.39 sec)   *****
// Found 1 benchmarks:
//   LinqBenchmarks.Count00LinqMethodX: Job-NKPLKD(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

// **************************
// Benchmark: LinqBenchmarks.Count00LinqMethodX: Job-NKPLKD(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
// *** Execute ***
// Launch: 1 / 1
// Execute: /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish/Job-NKPLKD --anonymousPipes 216 217 --benchmarkName LinqBenchmarks.Count00LinqMethodX --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1" --benchmarkId 0 in /home/helixbot/work/AE440975/w/B2600917/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-NKPLKD/bin/Release/net9.0/linux-x64/publish
// Failed to set up high priority (Permission denied). In order to run benchmarks with high priority, make sure you have the right permissions.
// BeforeAnythingElse

// Benchmark Process Environment Information:
// BenchmarkDotNet v0.13.13-nightly.20240213.132
// Runtime=.NET 9.0.0 (42.42.42.42424) using MonoVM, X64 X86Base
// GC=Non-concurrent Workstation
// HardwareIntrinsics=X86Base,LZCNT,POPCNT VectorSize=128
// Job: Job-OFMCXE(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

OverheadJitting  1: 1 op, 97937.00 ns, 97.9370 us/op
WorkloadJitting  1: 1 op, 802479139.00 ns, 802.4791 ms/op
...

This occurred with dotnet/performance commit hash 044ae265469659d6ef7bf65c8af040652fe7ef31, dotnet/runtime commit hash 9dc6ea62a4d195ae4559f4609a56933c61889756, and BenchmarkDotNet version 0.13.13-nightly.20240213.132.

I was able to repro the fail but still run by checking out the specified dotnet/performance repo hash, cd into the scripts folder, and running: sudo python3 benchmarks_local.py --commits 9dc6ea62a4d195ae4559f4609a56933c61889756 --run-types MonoAOTLLVM --filter *Span.IndexerBench.CoveredIndex2* *WriteReadAsync*. I did this in WSL on an Ubuntu 22.04.3 LTS instance.

Potentially related to https://github.com/dotnet/BenchmarkDotNet/issues/2311 and found during investigation of https://github.com/dotnet/runtime/issues/98967.

LoopedBard3 commented 9 months ago

FYI: @matouskozak

timcassell commented 9 months ago

I see, the change in #2367 returned error if it wasn't successful, while my refactor in #2393 just uses the ToBuildResult method, which handles it differently for warnings (I thought it was the same). https://github.com/dotnet/BenchmarkDotNet/blob/8631651a33f46c68e07972bbdf72eae7736fceb0/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommandResult.cs#L37-L40

That file is 5 years old at this point, is the exit code 1 from only a warning still an issue on newer sdks? Maybe we can remove that check. #2367 effectively did that anyway by skipping that check.