search

dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
701 stars 272 forks source link

Performance-ci centos tests failing in BDN #2496

Closed LoopedBard3 closed 2 years ago

LoopedBard3 commented 2 years ago

CentOS runs on the performance-ci are failing due to a type initializer exception. This can be seen in the pipeline here https://dev.azure.com/dnceng/public/_build/results?buildId=1838682&view=logs&jobId=c691a557-b8b7-5576-3a35-6adbad78c2b5, the full error is as follows:

[2022/06/22 12:09:18][INFO] System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property.
[2022/06/22 12:09:18][INFO]  ---> EETypeRva:0x030E3B60(System.Reflection.MissingRuntimeArtifactException): This object cannot be invoked because no code was generated for it: 'BenchmarkDotNet.Jobs.EnvironmentMode.PlatformCharacteristic'.
[2022/06/22 12:09:18][INFO]    at System.Reflection.Runtime.FieldInfos.RuntimeFieldInfo.get_FieldAccessor() + 0x11e
[2022/06/22 12:09:18][INFO]    at System.Reflection.Runtime.FieldInfos.RuntimeFieldInfo.GetValue(Object) + 0x10
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicHelper.<>c.<GetThisTypeCharacteristicsCore>b__8_1(FieldInfo) + 0x19
[2022/06/22 12:09:18][INFO]    at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() + 0x80
[2022/06/22 12:09:18][INFO]    at System.Linq.Enumerable.ConcatIterator`1.MoveNext() + 0x61
[2022/06/22 12:09:18][INFO]    at System.Collections.Generic.HashSet`1.UnionWith(IEnumerable`1) + 0x3f
[2022/06/22 12:09:18][INFO]    at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1, IEqualityComparer`1) + 0xfa
[2022/06/22 12:09:18][INFO]    at System.Linq.Enumerable.DistinctIterator`1.ToArray() + 0x36
[2022/06/22 12:09:18][INFO]    at System.Linq.Buffer`1..ctor(IEnumerable`1) + 0x34
[2022/06/22 12:09:18][INFO]    at System.Linq.OrderedEnumerable`1.ToArray() + 0x38
[2022/06/22 12:09:18][INFO]    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0x8d
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicHelper.FillAllCharacteristicsCore(Type, List`1, HashSet`1) + 0x28
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicHelper.GetAllCharacteristicsCore(Type) + 0x47
[2022/06/22 12:09:18][INFO]    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0x8d
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicObject.GetCharacteristicsToApply() + 0x2c
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicObject.GetCharacteristicsToApply(CharacteristicObject) + 0x1c
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Jobs.Job..cctor() + 0x204
[2022/06/22 12:09:18][INFO]    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xb9
[2022/06/22 12:09:18][INFO]    --- End of inner exception stack trace ---
[2022/06/22 12:09:18][INFO]    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0x153
[2022/06/22 12:09:18][INFO]    at System.Runtime.CompilerServices.ClassConstructorRunner.CheckStaticClassConstructionReturnGCStaticBase(StaticClassConstructionContext*, Object) + 0x9
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Jobs.Job..ctor(String) + 0x57
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost, String) + 0x1b4
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[]) + 0x256
LoopedBard3 commented 2 years ago

@adamsitnik @naricc Have y'all seen an error like this before with BDN? Here is a log with the error: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-performance-refs-heads-main-367830c2c6914864b5/Partition0/1/console.f1d1238a.log?helixlogtype=result Thanks!

adamsitnik commented 2 years ago

Hi @LoopedBard3 !

@MichalStrehovsky has recently fixed this issue with https://github.com/dotnet/BenchmarkDotNet/pull/2020

I've just fixed a recent breaking change: https://github.com/dotnet/BenchmarkDotNet/pull/2045

To fix these builds we need is to upload 0.13.1.1818 to our internal feed and update BDN version in perf repo. Could you please do that?

BTW BenchmarkDotNet has recently updated all it's dependencies like TraceEvent, ClrMd and Iced. Some of them might not be in our internal feed(s) yet and you might need to upload them as well.

LoopedBard3 commented 2 years ago

Taking a look!

LoopedBard3 commented 2 years ago

It looks like there is a new 0.13.1.18.19 version, any reason not to take that one?

MichalStrehovsky commented 2 years ago

0.13.1.1820 should have my fix. Can we try with that one?

adamsitnik commented 2 years ago

@MichalStrehovsky I've verified that your fix has solved this problem in https://github.com/dotnet/performance/pull/2534, but a new one popped out: https://github.com/dotnet/roslyn/issues/62864

@LoopedBard3 could you please update to 0.13.1.1820? It's going to help and then as soon as https://github.com/dotnet/roslyn/issues/62864 gets fixed the CI should become green again

MichalStrehovsky commented 2 years ago

@MichalStrehovsky I've verified that your fix has solved this problem in https://github.com/dotnet/performance/pull/2534, but a new one popped out: https://github.com/dotnet/roslyn/issues/62864

If it's blocking, downgrading the Roslyn used to build this repo to a build from 2 weeks ago would work around.

LoopedBard3 commented 2 years ago

Updated to 0.13.1.1823 in the #2532 PR.

LoopedBard3 commented 2 years ago

Looks like the BDN changes are running properly, but there are some tests that are unsupported:

[2022/07/22 15:50:58][INFO] // Found 1 benchmarks:
[2022/07/22 15:50:58][INFO] //   MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // **************************
[2022/07/22 15:50:58][INFO] // Benchmark: MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] // *** Execute ***
[2022/07/22 15:50:58][INFO] // Launch: 1 / 1
[2022/07/22 15:50:58][INFO] // Execute: /home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/91d5ac24-fe26-461f-afdd-5e7aeda1a89d/bin/Release/net7.0/linux-x64/publish/91d5ac24-fe26-461f-afdd-5e7aeda1a89d --benchmarkName "BenchmarksGame.MandelBrot_7.Bench(size: 4000, lineLength: 500, checksum: \"C7-E6-66-43-66-73-F8-A8-D3-B4-D7-97-2F-FC-A1-D3\")" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0" --benchmarkId 9 in
[2022/07/22 15:50:58][INFO] Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
[2022/07/22 15:50:58][INFO] // BeforeAnythingElse
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // Benchmark Process Environment Information:
[2022/07/22 15:50:58][INFO] // Runtime=.NET 7.0.0-rc.1.22372.1, X64 NativeAOT
[2022/07/22 15:50:58][INFO] // GC=Non-concurrent Workstation
[2022/07/22 15:50:58][INFO] // Job: Job-UDTYPC(PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // BeforeActualRun
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] System.PlatformNotSupportedException: Operation is not supported on this platform.
[2022/07/22 15:50:58][INFO]    at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowPlatformNotSupportedException() + 0x20
[2022/07/22 15:50:58][INFO]    at BenchmarksGame.MandelBrot_7.DoBench(Int32, Int32) + 0x8a
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_9.WorkloadActionUnroll(Int64) + 0x4d
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData) + 0x1d3
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.EngineStage.RunIteration(IterationMode, IterationStage, Int32, Int64, Int32) + 0x87
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.EngineActualStage.RunSpecific(Int64, IterationMode, Int32, Int32) + 0x79
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.Engine.Run() + 0x291
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_9.Run(IHost, String) + 0x6af
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[]) + 0x2ee
[2022/07/22 15:50:58][INFO] // AfterAll
[2022/07/22 15:50:58][INFO] No Workload Results were obtained from the run.
[2022/07/22 15:50:58][INFO] // Benchmark Process 9718 has exited with code 255.
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // ** Remained 852 (98.8%) benchmark(s) to run. Estimated finish 2022-07-22 16:13 (0h 22m from now) **
[2022/07/22 15:50:58][INFO] // ***** BenchmarkRunner: Finish  *****
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Export *
[2022/07/22 15:50:58][INFO]   home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/BenchmarkDotNet.Artifacts/results/BenchmarksGame.MandelBrot_7-report-github.md
[2022/07/22 15:50:58][INFO]   home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/BenchmarkDotNet.Artifacts/results/BenchmarksGame.MandelBrot_7-report-full.json
[2022/07/22 15:50:58][INFO]   home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/BenchmarkDotNet.Artifacts/results/BenchmarksGame.MandelBrot_7-perf-lab-report.json
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Detailed results *
[2022/07/22 15:50:58][INFO] MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] Runtime = .NET 7.0.0-rc.1.22372.1, X64 NativeAOT; GC = Non-concurrent Workstation
[2022/07/22 15:50:58][INFO] There are not any results runs
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Summary *
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] BenchmarkDotNet=v0.13.1.1823-nightly, OS=centos 7
[2022/07/22 15:50:58][INFO] AMD EPYC 7452, 1 CPU, 2 logical cores and 1 physical core
[2022/07/22 15:50:58][INFO] .NET SDK=7.0.100-rc.1.22371.5
[2022/07/22 15:50:58][INFO]   [Host]     : .NET 7.0.0 (7.0.22.36704), X64 RyuJIT
[2022/07/22 15:50:58][INFO]   Job-CLDRFN : .NET 7.0.0-rc.1.22372.1, X64 NativeAOT
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] PowerPlanMode=00000000-0000-0000-0000-000000000000  Runtime=NativeAOT 7.0  Toolchain=ILCompiler 7.0.0-*
[2022/07/22 15:50:58][INFO] InvocationCount=1  IterationCount=1  IterationTime=250.0000 ms
[2022/07/22 15:50:58][INFO] MaxIterationCount=20  MinIterationCount=15  RunStrategy=ColdStart
[2022/07/22 15:50:58][INFO] UnrollFactor=1  WarmupCount=0
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] |       Method | size | lineLength |                             checksum | Mean | Error | Median | Min | Max |
[2022/07/22 15:50:58][INFO] |------------- |----- |----------- |------------------------------------- |-----:|------:|-------:|----:|----:|
[2022/07/22 15:50:58][INFO] | MandelBrot_7 | 4000 |        500 | C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47] |   NA |    NA |     NA |  NA |  NA |
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] Benchmarks with issues:
[2022/07/22 15:50:58][INFO]   MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Legends *
[2022/07/22 15:50:58][INFO]   size       : Value of the 'size' parameter
[2022/07/22 15:50:58][INFO]   lineLength : Value of the 'lineLength' parameter
[2022/07/22 15:50:58][INFO]   checksum   : Value of the 'checksum' parameter
[2022/07/22 15:50:58][INFO]   Mean       : Arithmetic mean of all measurements
[2022/07/22 15:50:58][INFO]   Error      : Half of 99.9% confidence interval
[2022/07/22 15:50:58][INFO]   Median     : Value separating the higher half of all measurements (50th percentile)
[2022/07/22 15:50:58][INFO]   Min        : Minimum
[2022/07/22 15:50:58][INFO]   Max        : Maximum
[2022/07/22 15:50:58][INFO]   1 ns       : 1 Nanosecond (0.000000001 sec)
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Diagnostic Output - MemoryDiagnoser *
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // ***** BenchmarkRunner: End *****
[2022/07/22 15:50:58][INFO] Run time: 00:00:00 (0.04 sec), executed benchmarks: 1

If y'all see something immediately obvious as to why this error was hit (pertaining to what isn't supported, not that something is not supported), please respond.

MichalStrehovsky commented 2 years ago

If y'all see something immediately obvious as to why this error was hit

It's the issue Adam wrote about above

LoopedBard3 commented 2 years ago

It appears that the performance-ci tests for centos are now succeeding on main after merging https://github.com/dotnet/performance/pull/2532. Closing issue.