search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.43k stars 960 forks source link

Bug. Native AOT .NET 7.0 doesn't work. System.NotSupportedException: X86Serialize #2463

Closed Avangardum closed 10 months ago

Avangardum commented 10 months ago

I created a very simple benchmark:

[SimpleJob(RuntimeMoniker.NativeAot70)]
public class Benchmarks
{
    [Benchmark]
    public void Foo()
    {
        Thread.Sleep(5);
    }
}

When I run it, I get the following output:

"C:\Program Files\JetBrains\JetBrains Rider 2023.1.2\plugins\dpa\DotFiles\JetBrains.DPA.Runner.exe" --handle=26532 --backend-pid=15812 --etw-collect-flags=3 --detach-event-name=dpa.detach.26532 C:/Users/Yuriy/RiderProjects/BenchmarkDotNetBugReprodution/BenchmarkDotNetBugReproduction/bin/Release/net7.0/BenchmarkDotNetBugReproduction.exe
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 1 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet  restore -r win-x64 /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9
// command took 1.02 sec and exited with 0
// start dotnet  build -c Release -r win-x64 --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9
// command took 3.55 sec and exited with 0
// start dotnet  publish -c Release -r win-x64 --no-build --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true --output "C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9\bin\Release\net7.0\win-x64\publish" in C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9
// command took 2.9 sec and exited with 1
// ***** Done, took 00:00:07 (7.59 sec)   *****
// Found 1 benchmarks:
//   Benchmarks.Foo: NativeAOT 7.0(Runtime=NativeAOT 7.0)

Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// Build Error: Standard output: 

 Standard error:
 MSBuild version 17.7.3+4fca21998 for .NET
C:\Program Files\dotnet\sdk\7.0.403\Sdks\Microsoft.NET.ILLink.Tasks\build\Microsoft.NET.ILLink.targets(205,5): warning : Property 'TrimmerDefaultAction' is deprecated in .NET 7 and will be ignored. Use TrimMode instead. [C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9\BenchmarkDotNet.Autogenerated.csproj]
  Generating native code
EXEC : error : X86Serialize [C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9\BenchmarkDotNet.Autogenerated.csproj]
  System.NotSupportedException: X86Serialize
     at ILCompiler.HardwareIntrinsicHelpers.XArchIntrinsicConstants.FromInstructionSet(InstructionSet) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/HardwareIntrinsicHelpers.Aot.cs:line 153
     at ILCompiler.HardwareIntrinsicHelpers.GetRuntimeRequiredIsaFlags(InstructionSetSupport) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/HardwareIntrinsicHelpers.Aot.cs:line 70
     at ILCompiler.ExpectedIsaFeaturesRootProvider.ILCompiler.ICompilationRootProvider.AddCompilationRoots(IRootingServiceProvider) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/ExpectedIsaFeaturesRootProvider.cs:line 27
     at ILCompiler.Compilation..ctor(DependencyAnalyzerBase`1, NodeFactory, IEnumerable`1, ILProvider, DebugInformationProvider, DevirtualizationManager, IInliningPolicy, Logger) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Compilation.cs:line 64
     at ILCompiler.ILScannerBuilder.ToILScanner() in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/ILScannerBuilder.cs:line 83
     at ILCompiler.Program.<Run>g__RunScanner|73_4(<>c__DisplayClass73_0&) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 884
     at ILCompiler.Program.Run(String[]) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 866
     at ILCompiler.Program.Main(String[]) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 1150
C:\Program Files\dotnet\sdk\7.0.403\Sdks\Microsoft.DotNet.ILCompiler\build\Microsoft.NETCore.Native.targets(278,5): error MSB3073: The command ""C:\Users\Yuriy\.nuget\packages\runtime.win-x64.microsoft.dotnet.ilcompiler\7.0.13\t
ools\\ilc" @"obj\Release\net7.0\win-x64\native\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9.ilc.rsp"" exited with code 1. [C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9\BenchmarkDotNet.Autogenerated.csproj]

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Users\Yuriy\RiderProjects\BenchmarkDotNetBugReprodution\BenchmarkDotNetBugReproduction\bin\Release\net7.0\6a5ac22a-b1e1-4db0-8019-a0b40ad5fdb9
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

// ** Remained 0 (0.0%) benchmark(s) to run. Estimated finish 2023-11-10 13:27 (0h 0m from now) **
Successfully reverted power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNetBugReproduction.Benchmarks-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNetBugReproduction.Benchmarks-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNetBugReproduction.Benchmarks-report.html

// * Detailed results *
Benchmarks.Foo: NativeAOT 7.0(Runtime=NativeAOT 7.0)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.2428/22H2/2022Update/SunValley2)
12th Gen Intel Core i7-12700H, 1 CPU, 20 logical and 14 physical cores
.NET SDK 7.0.403
  [Host] : .NET 7.0.13 (7.0.1323.51816), X64 RyuJIT AVX2

Job=NativeAOT 7.0  Runtime=NativeAOT 7.0

| Method | Mean | Error |
|------- |-----:|------:|
| Foo    |   NA |    NA |

Benchmarks with issues:
  Benchmarks.Foo: NativeAOT 7.0(Runtime=NativeAOT 7.0)

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
Run time: 00:00:00 (0.06 sec), executed benchmarks: 0

Global total time: 00:00:08 (8.03 sec), executed benchmarks: 0
// * Artifacts cleanup *
Artifacts cleanup is finished

Process finished with exit code 0.
adamsitnik commented 10 months ago

Hi @Avangardum

Could you please share your project file?

Avangardum commented 10 months ago

Hello, @adamsitnik.

I uploaded the whole solution https://github.com/Avangardum/BenchmarkDotNetBugReproduction

adamsitnik commented 10 months ago

@Avangardum thanks for sharing the repro.

It works on my machine:

BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.2428/22H2/2022Update/SunValley2)
AMD Ryzen Threadripper PRO 3945WX 12-Cores, 1 CPU, 24 logical and 12 physical cores
.NET SDK 9.0.100-alpha.1.23531.2
  [Host]        : .NET 7.0.13 (7.0.1323.51816), X64 RyuJIT AVX2
  NativeAOT 7.0 : .NET 7.0.11-servicing.23424.27, X64 NativeAOT AVX2

Job=NativeAOT 7.0  Runtime=NativeAOT 7.0

| Method | Mean     | Error    | StdDev   |
|------- |---------:|---------:|---------:|
| Foo    | 16.56 ms | 0.171 ms | 0.160 ms |

Could you please apply the following attribute:

[SimpleJob(RuntimeMoniker.NativeAot70)]
+ [KeepBenchmarkFilesAttribute]
public class Benchmarks

And share the "IlcInstructionSet" part of BenchmarkDotNetBugReproduction\bin\Release\net7.0\NativeAOT 7.0\BenchmarkDotNet.Autogenerated.csproj

Here is mine:

<IlcInstructionSet>base,sse,sse2,sse3,ssse3,sse4.1,sse4.2,avx,avx2,aes,bmi,bmi2,fma,lzcnt,pclmul,popcnt</IlcInstructionSet>
Avangardum commented 10 months ago 
<IlcInstructionSet>base,sse,sse2,sse3,ssse3,sse4.1,sse4.2,avx,avx2,aes,bmi,bmi2,fma,lzcnt,pclmul,popcnt,avxvnni,serialize</IlcInstructionSet>
adamsitnik commented 10 months ago

@MichalStrehovsky Do you have any idea why using X86Serialize fails with an exception for @Avangardum ? It works fine on my machine, however I am using newer SDK.

MichalStrehovsky commented 10 months ago

https://github.com/dotnet/BenchmarkDotNet/issues/2463#issuecomment-1808495018 has serialize as an instruction set specified. This cannot be set on .NET 7 due to a bug (whoever was adding the instruction set forgot to add AOT handling). This bug was fixed in https://github.com/dotnet/runtime/commit/c81e83ab01fb3aed44de34e5aff659d28e6c91f4#diff-65f20fbb1fbd6c815168e9d3b2b358c4fd02aea226f2152099427a594310b876R114. It's fixed in .NET 8.

adamsitnik commented 10 months ago

@Avangardum the bug got fixed, you can either wait until 0.13.11 gets shipped to nuget.org (2 weeks from now) or try latest version from our preview feed: https://benchmarkdotnet.org/articles/guides/nuget.html?q=nuget#nightly