.NET 7.0 RC2 Microbenchmarks Performance Study Report

jozkee commented 1 year ago

Data

This time we have covered following configurations comparing .NET 7.0 RC2 vs. .NET 6.0:

architectures: arm64, x64
Unix: debian 11, ubuntu 18.04, ubuntu 20.04, macOS Monterey 12.6, macOS Big Sur 11.7
Windows: 10 & 11

| Operating System | Bit | Processor Name | | ------------------- | ----- | --------------------------------------------- | | ubuntu 18.04 | Arm64 | Unknown processor | | Windows 11 | Arm64 | Unknown processor | | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | macOS Monterey 12.6 | Arm64 | Apple M1 | | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | Windows 11 | X64 | AMD Ryzen 9 5900X | | Windows 11 | X64 | AMD Ryzen 9 7950X | | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) |

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @adamsitnik, @brianrob, @carlossanlop, @dakersnar, @janvorli, @jeffhandley, @cincuranet, @tannergooding, and @wfurt who contributed their results and time. An addtional thank you to @dakersnar and @EgorBo for shadowing the creation of this report and helping with the vetting of the results!

The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read https://github.com/dotnet/runtime/issues/41871. RC1 report can be found here.

Improvements

We will be analyzing improvements more thoroughly in the upcoming .NET 6 vs .NET 7 GA report. This report is focused on regressions.

Regressions

By Design

[x] Benchstone.BenchI.Array2.Test
- Detected by: https://github.com/dotnet/runtime/issues/71059
- Explained by: https://github.com/dotnet/runtime/issues/71059#issuecomment-1203196867
- This regression was included in .NET 7.0 Preview 7 report https://github.com/dotnet/runtime/issues/73866.
[x] Benchstone.MDBenchI.MDMulMatrix.Test
- Detected by: https://github.com/dotnet/runtime/issues/72030
- Explained by: https://github.com/dotnet/runtime/issues/72030#issuecomment-1213512373
- This regression was included in .NET 7.0 Preview 7 report https://github.com/dotnet/runtime/issues/73866.
- Note: We noticed arm64 improvements despite x64 regressions.
[x] System.Collections.Concurrent.IsEmpty<String>.Dictionary(Size: 512), System.Collections.Concurrent.IsEmpty<Int32>.Dictionary(Size: 512)
- Introduced by https://github.com/dotnet/runtime/pull/73488 -
- Explained by: https://github.com/dotnet/runtime/pull/73488#issuecomment-1245922399
- This regression was included in .NET 7.0 RC1 report https://github.com/dotnet/runtime/issues/76320.
[x] System.Collections.IndexerSet<String>.Array(Size: 512)
- Detected by: https://github.com/dotnet/perf-autofiling-issues/issues/6746
- Added comment: https://github.com/dotnet/runtime/pull/71659#issuecomment-1278221030
[x] System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False)), System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False)), System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False)), System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))
- Explained by: https://github.com/dotnet/runtime/pull/69878
- Added comment: https://github.com/dotnet/runtime/pull/69878#issuecomment-1281528243
- This regression was incldued in .NET 7.0 preview 7 report https://github.com/dotnet/runtime/pull/69878.
[x] System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt"), System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf")
- Detected by: https://github.com/dotnet/runtime/issues/73391
- Explained by: https://github.com/dotnet/runtime/issues/73391#issuecomment-1205619431
- Note: Perf test bug, will be fixed in https://github.com/dotnet/performance/pull/2567. Already discussed in .NET 7.0 RC1 report https://github.com/dotnet/runtime/issues/76320.
[x] System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]), System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]), System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 0, 0]), System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 1, 1])
- Explained by: https://github.com/dotnet/runtime/pull/60463#issuecomment-945828201
- For all other cases, the perf drastically improved.
[x] System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits), System.Numerics.Tests.Perf_BigInteger.Add(arguments: 65536,65536 bits)
- Regression was reported by the bot (https://github.com/dotnet/runtime/issues/60293), but not for this particular argument
- Added comment: https://github.com/dotnet/runtime/issues/60293#issuecomment-1072434031
- Explained by: https://github.com/dotnet/runtime/issues/60293#issuecomment-1083543708
- This regression was included in .NET 7.0 Preview 2 report https://github.com/dotnet/runtime/issues/66848.
[x] System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark, System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark
- Introduced by: https://github.com/dotnet/runtime/pull/68691
[x] System.Numerics.Tests.Perf_VectorConvert.Convert_double_long:
- Tracked by: https://github.com/dotnet/runtime/issues/65189
- Explained by: https://github.com/dotnet/runtime/issues/65189#issuecomment-1036388904
- This regression was included in .NET 7.0 preview 1 report https://github.com/dotnet/runtime/issues/65200
- Note: As per @tannergooding, this regression can be fixed.
[x] System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes.DeriveBytes
- specific to Unix, Windows seems to be unaffected
- Explained by: https://github.com/dotnet/runtime/issues/65719#issuecomment-1048339034
- This regression was included in .NET 7.0 preview 2 report https://github.com/dotnet/runtime/issues/66848
[x] System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?m)^Sherlock Holmes|Sherlock Holmes$", Options: Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)
- Explained by: https://github.com/dotnet/runtime/issues/65187#issuecomment-1036287653
- Note: While we were looking at the results, we noticed a mono-specific regression, created issue https://github.com/dotnet/runtime/issues/77129.
[x] System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: None)
- Explained by: https://github.com/dotnet/runtime/issues/67915#issuecomment-1096979972
- This regression was included in .NET 7.0 Preview 4 report https://github.com/dotnet/runtime/issues/69396.

Investigation In Progress

[x] Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty
- Detected by: https://github.com/dotnet/runtime/issues/77900
- Added comment: https://github.com/dotnet/runtime/issues/77900
[ ] FractalPerf.Launch.Test
- Detected by: https://github.com/dotnet/perf-autofiling-issues/issues/4414, but not solved
- the group of benchmark regressions in the above issue were addressed in https://github.com/dotnet/runtime/issues/67594
- Explained by: https://github.com/dotnet/runtime/issues/67594#issuecomment-1089180924
- Added comment as this went unresolved since preview 4 report: https://github.com/dotnet/runtime/issues/67594#issuecomment-1278172793
[ ] PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod, System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1]), System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128]), System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024]), System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0]), System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024], System.Text.Tests.Perf_Encoding.GetEncoder*
- Detected by: https://github.com/dotnet/runtime/issues/74014.
- Explained by: https://github.com/dotnet/runtime/issues/74014#issuecomment-1230319972
- Added comments: https://github.com/dotnet/runtime/issues/74014#issuecomment-1278190959
- Note: GC related.
[ ] System.Collections.Sort<BigStruct>.Array_ComparerClass(Size: 512), System.Collections.Sort<BigStruct>.Array_ComparerStruct(Size: 512), System.Collections.Sort<BigStruct>.Array_Comparison(Size: 512), System.Collections.Sort<BigStruct>.LinqQuery(Size: 512).
- Seems like noise on macOS and we thought that it would be good to ask for a re-run to the contributor.
[ ] System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateGetPositionArray
- Created issue: https://github.com/dotnet/runtime/issues/77028
- Note: Gradual regression detected over .NET 7 release cycle.
[x] System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: List)
- Added comment: https://github.com/dotnet/runtime/pull/65738#issuecomment-1279259957
[ ] System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark , System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark, System.Numerics.Tests.Perf_Matrix4x4.Transpose, System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark, System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark
- Tracked here, added comment: https://github.com/dotnet/runtime/issues/65191#issuecomment-1279304302
- Already included in .NET 7.0 Preview 1 https://github.com/dotnet/runtime/issues/65200.
[ ] System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True), System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)
- Added comment: https://github.com/dotnet/runtime/issues/65198#issuecomment-1279474550
- For non-macOS configurations, historical data indicates that this regression is solved already. We don't have historical data for Mac machines, so this is technically under investigation.
[x] System.Tests.Perf_String.CtorCharCount(size: 1), System.Tests.Perf_String.CtorCharCount(size: 10), System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8), System.Text.Tests.Perf_StringBuilder.ctor_capacity(length: 100000), System.Tests.Perf_String.Trim(s: " Test")
- Added comment: https://github.com/dotnet/runtime/issues/77064#issuecomment-1279505432
[x] System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic), System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
- Detected by: https://github.com/dotnet/runtime/issues/68405
- Added comment: https://github.com/dotnet/runtime/issues/68405#issuecomment-1279535304

Noise, Flaky or Multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

[x] System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits)
- We are seeing 15-25% regressions in the report data but this doesn't match the historical data.
[x] System.Numerics.Tests.Perf_Matrix4x4.LerpBenchmark
[x] System.Numerics.Tests.Perf_VectorOf<*>.*

Statistics

Total: 86862 Same: 55.57 % Slower: 7.85 % Faster: 23.65 % Noise: 12.88 % Unknown: 0.04 %

Statistics per Architecture

Architecture	Same	Slower	Faster	Noise	Unknown
Arm64	50.16 %	2.66 %	34.81 %	12.31 %	0.05 %
X64	58.07 %	10.24 %	18.51 %	13.15 %	0.04 %

Statistics per Operating System

Operating System	Same	Slower	Faster	Noise	Unknown
macOS Monterey 12.6	49.99 %	2.60 %	34.20 %	13.16 %	0.05 %
ubuntu 18.04	61.50 %	6.66 %	21.50 %	10.25 %	0.09 %
Windows 11	57.64 %	6.26 %	22.91 %	13.15 %	0.03 %
debian 11	63.41 %	7.07 %	16.30 %	13.17 %	0.04 %
macOS Big Sur 11.7	63.45 %	3.35 %	19.18 %	13.94 %	0.07 %
ubuntu 20.04	49.65 %	13.46 %	22.58 %	14.29 %	0.01 %
Windows 10	42.13 %	26.62 %	18.84 %	12.42 %	0.00 %

Statistics per Namespace

| Namespace | Same | Slower | Faster | Noise | Unknown| | --------------------------------------------------------- | ------- | ------- | ------- | ------- | ------- | | Benchmark | 56.58 % | 32.89 % | 10.53 % | 0.00 % | 0.00 % | | BenchmarksGame | 65.41 % | 8.52 % | 26.07 % | 0.00 % | 0.00 % | | Benchstone.BenchF | 76.84 % | 6.05 % | 17.11 % | 0.00 % | 0.00 % | | Benchstone.BenchI | 53.95 % | 10.09 % | 35.96 % | 0.00 % | 0.00 % | | Benchstone.MDBenchF | 42.11 % | 3.16 % | 54.74 % | 0.00 % | 0.00 % | | Benchstone.MDBenchI | 20.00 % | 9.47 % | 70.53 % | 0.00 % | 0.00 % | | Devirtualization | 65.26 % | 2.11 % | 1.05 % | 31.58 % | 0.00 % | | Exceptions | 77.05 % | 1.32 % | 21.64 % | 0.00 % | 0.00 % | | FractalPerf | 5.26 % | 89.47 % | 5.26 % | 0.00 % | 0.00 % | | GuardedDevirtualization | 69.42 % | 0.00 % | 4.51 % | 26.07 % | 0.00 % | | IfStatements | 33.68 % | 0.00 % | 66.32 % | 0.00 % | 0.00 % | | Inlining | 33.33 % | 0.00 % | 1.75 % | 64.91 % | 0.00 % | | Interop | 89.47 % | 0.00 % | 10.53 % | 0.00 % | 0.00 % | | JetStream | 34.21 % | 2.63 % | 63.16 % | 0.00 % | 0.00 % | | Layout | 84.21 % | 0.00 % | 15.79 % | 0.00 % | 0.00 % | | Lowering | 84.21 % | 0.00 % | 0.00 % | 15.79 % | 0.00 % | | MicroBenchmarks.Serializers | 62.27 % | 3.75 % | 33.98 % | 0.00 % | 0.00 % | | Microsoft.AspNetCore.Server.Kestrel.Performance | 63.16 % | 10.53 % | 26.32 % | 0.00 % | 0.00 % | | Microsoft.Extensions.Caching.Memory.Tests | 69.38 % | 13.88 % | 16.75 % | 0.00 % | 0.00 % | | Microsoft.Extensions.Configuration | 65.50 % | 25.15 % | 9.36 % | 0.00 % | 0.00 % | | Microsoft.Extensions.Configuration.Xml | 65.79 % | 5.26 % | 28.95 % | 0.00 % | 0.00 % | | Microsoft.Extensions.DependencyInjection | 45.67 % | 3.57 % | 48.90 % | 1.87 % | 0.00 % | | Microsoft.Extensions.Logging | 64.66 % | 3.16 % | 28.42 % | 3.76 % | 0.00 % | | Microsoft.Extensions.Primitives.Performance | 50.00 % | 0.66 % | 1.32 % | 48.03 % | 0.00 % | | Microsoft.Extensions.Primitives | 42.40 % | 0.29 % | 17.84 % | 39.47 % | 0.00 % | | PerfLabTests | 56.18 % | 12.70 % | 24.47 % | 6.64 % | 0.00 % | | PerfLabTests.CastingPerf2 | 49.12 % | 14.04 % | 36.84 % | 0.00 % | 0.00 % | | SciMark2 | 64.21 % | 1.05 % | 34.74 % | 0.00 % | 0.00 % | | SIMD | 90.06 % | 0.58 % | 9.36 % | 0.00 % | 0.00 % | | Span | 79.79 % | 0.42 % | 19.79 % | 0.00 % | 0.00 % | | StoreBlock | 52.81 % | 0.00 % | 4.04 % | 43.16 % | 0.00 % | | System.Buffers.Binary.Tests | 43.42 % | 4.61 % | 25.66 % | 26.32 % | 0.00 % | | System.Buffers.Tests | 54.36 % | 10.93 % | 28.04 % | 6.67 % | 0.00 % | | System.Buffers.Text.Tests | 48.22 % | 3.79 % | 21.52 % | 26.47 % | 0.00 % | | System.Collections | 63.24 % | 7.83 % | 27.74 % | 1.19 % | 0.00 % | | System.Collections.Concurrent | 67.46 % | 7.75 % | 12.96 % | 11.83 % | 0.00 % | | System.Collections.Tests | 53.76 % | 5.46 % | 28.87 % | 11.90 % | 0.00 % | | System.ComponentModel.Tests | 38.76 % | 0.48 % | 60.77 % | 0.00 % | 0.00 % | | System.ConsoleTests | 69.30 % | 8.77 % | 21.93 % | 0.00 % | 0.00 % | | System.Diagnostics | 49.47 % | 7.11 % | 43.42 % | 0.00 % | 0.00 % | | System.Diagnostics.Tracing | 75.00 % | 9.21 % | 15.79 % | 0.00 % | 0.00 % | | System.Formats.Cbor.Tests | 65.99 % | 4.05 % | 29.96 % | 0.00 % | 0.00 % | | System.Globalization.Tests | 75.37 % | 6.03 % | 15.79 % | 2.81 % | 0.00 % | | System | 65.20 % | 3.51 % | 28.07 % | 3.22 % | 0.00 % | | System.IO.Compression | 80.92 % | 3.29 % | 15.79 % | 0.00 % | 0.00 % | | System.IO.MemoryMappedFiles.Tests | 26.36 % | 4.09 % | 69.55 % | 0.00 % | 0.00 % | | System.IO.Pipelines.Tests | 69.74 % | 1.32 % | 28.95 % | 0.00 % | 0.00 % | | System.IO.Pipes.Tests | 73.68 % | 16.67 % | 9.65 % | 0.00 % | 0.00 % | | System.IO.Tests | 60.55 % | 12.12 % | 24.61 % | 2.72 % | 0.00 % | | System.Linq.Tests | 77.64 % | 7.45 % | 14.17 % | 0.73 % | 0.00 % | | System.MathBenchmarks | 87.83 % | 1.81 % | 8.47 % | 1.89 % | 0.00 % | | System.Memory | 51.32 % | 1.89 % | 19.58 % | 27.21 % | 0.00 % | | System.Net.Http.Tests | 42.11 % | 31.58 % | 26.32 % | 0.00 % | 0.00 % | | System.Net.NetworkInformation.Tests | 42.11 % | 19.74 % | 38.16 % | 0.00 % | 0.00 % | | System.Net.Primitives.Tests | 60.29 % | 9.57 % | 15.55 % | 14.59 % | 0.00 % | | System.Net.Security.Tests | 74.10 % | 14.43 % | 11.48 % | 0.00 % | 0.00 % | | System.Net.Sockets.Tests | 49.45 % | 23.08 % | 27.47 % | 0.00 % | 0.00 % | | System.Net.Tests | 71.56 % | 13.76 % | 14.68 % | 0.00 % | 0.00 % | | System.Net.WebSockets.Tests | 47.37 % | 39.47 % | 13.16 % | 0.00 % | 0.00 % | | System.Numerics.Tests | 44.69 % | 5.24 % | 6.04 % | 44.03 % | 0.00 % | | System.Reflection | 30.50 % | 2.15 % | 64.35 % | 2.99 % | 0.00 % | | System.Runtime.InteropServices.Tests | 78.95 % | 2.63 % | 15.79 % | 2.63 % | 0.00 % | | System.Runtime.Serialization.Formatters.Tests | 42.11 % | 52.63 % | 5.26 % | 0.00 % | 0.00 % | | System.Security.Cryptography.Primitives.Tests.Performance | 84.87 % | 9.21 % | 5.92 % | 0.00 % | 0.00 % | | System.Security.Cryptography.Tests | 69.59 % | 7.02 % | 23.39 % | 0.00 % | 0.00 % | | System.Security.Cryptography.X509Certificates.Tests | 73.68 % | 12.28 % | 14.04 % | 0.00 % | 0.00 % | | System.Tests | 50.35 % | 12.24 % | 25.46 % | 11.95 % | 0.00 % | | System.Text.Encodings.Web.Tests | 72.59 % | 14.47 % | 11.84 % | 1.10 % | 0.00 % | | System.Text.Json.Document.Tests | 60.21 % | 10.53 % | 29.26 % | 0.00 % | 0.00 % | | System.Text.Json.Reader.Tests | 46.05 % | 15.79 % | 38.16 % | 0.00 % | 0.00 % | | System.Text.Json.Serialization.Tests | 57.60 % | 11.37 % | 31.03 % | 0.00 % | 0.00 % | | System.Text.Json.Tests | 67.98 % | 11.07 % | 20.59 % | 0.35 % | 0.00 % | | System.Text | 63.51 % | 24.21 % | 12.28 % | 0.00 % | 0.00 % | | System.Text.RegularExpressions.Tests | 16.73 % | 5.44 % | 76.91 % | 0.00 % | 0.92 % | | System.Text.Tests | 57.52 % | 21.93 % | 18.67 % | 1.88 % | 0.00 % | | System.Threading.Channels.Tests | 54.39 % | 17.54 % | 28.07 % | 0.00 % | 0.00 % | | System.Threading.Tasks.Tests | 47.37 % | 19.30 % | 33.33 % | 0.00 % | 0.00 % | | System.Threading.Tasks | 69.55 % | 17.67 % | 11.65 % | 1.13 % | 0.00 % | | System.Threading.Tests | 58.90 % | 11.31 % | 17.13 % | 12.65 % | 0.00 % | | System.Xml.Linq | 59.00 % | 9.70 % | 22.16 % | 9.14 % | 0.00 % | | System.Xml.Tests | 5.26 % | 2.11 % | 92.63 % | 0.00 % | 0.00 % | | V8.Crypto | 78.95 % | 15.79 % | 5.26 % | 0.00 % | 0.00 % | | V8.Richards | 73.68 % | 5.26 % | 21.05 % | 0.00 % | 0.00 % | | XmlDocumentTests.XmlDocumentTests | 40.35 % | 22.81 % | 21.05 % | 15.79 % | 0.00 % | | XmlDocumentTests.XmlNodeListTests | 86.84 % | 5.26 % | 2.63 % | 5.26 % | 0.00 % | | XmlDocumentTests.XmlNodeTests | 36.84 % | 2.63 % | 0.00 % | 60.53 % | 0.00 % | | System.Drawing.Tests | 77.50 % | 1.87 % | 15.00 % | 5.63 % | 0.00 % | | HardwareIntrinsics.RayTracer | 76.92 % | 7.69 % | 15.38 % | 0.00 % | 0.00 % |

Big thanks to everyone involved!

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.

Issue Details

## Data This time we have covered following configurations comparing .NET 7.0 RC2 vs. .NET 6.0: * architectures: arm64, x64 * Unix: debian 11, ubuntu 18.04, ubuntu 20.04, macOS Monterey 12.6, macOS Big Sur 11.7 * Windows: 10 & 11

| Operating System | Bit | Processor Name | | ------------------- | ----- | --------------------------------------------- | | ubuntu 18.04 | Arm64 | Unknown processor | | Windows 11 | Arm64 | Unknown processor | | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | macOS Monterey 12.6 | Arm64 | Apple M1 | | macOS Monterey 12.6 | Arm64 | Apple M1 Max | | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | Windows 11 | X64 | AMD Ryzen 9 5900X | | Windows 11 | X64 | AMD Ryzen 9 7950X | | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | debian 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | ubuntu 18.04 | X64 | AMD Ryzen 9 5900X | | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | ubuntu 20.04 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | ubuntu 20.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | macOS Big Sur 11.7 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | macOS Monterey 12.6 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) |

Most of the benchmarks were run on bare-metal machines, some were executed via WSL. This would not be possible without the help from: @adamsitnik, @brianrob, @carlossanlop, @dakersnar, @janvorli, @jeffhandley, @cincuranet, @tannergooding, and @wfurt who contributed their results and time. An addtional thank you to @dakersnar and @EgorBo for shadowing the creation of this report and helping with the vetting of the results! The full report generated by the tool is available [here](https://gist.github.com/Jozkee/54d7e86adf16381b32ce7178e0435618). You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them! Again, the [full historical data](https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/TestHistoryIndexIndex.html) turned out to be **extremely useful**. For details about methodology please read https://github.com/dotnet/runtime/issues/41871. RC1 report can be found [here](https://github.com/dotnet/runtime/issues/76320). ## Improvements We will be analyzing improvements more thoroughly in the upcoming .NET 6 vs .NET 7 GA report. This report is focused on regressions. ## Regressions ### By Design * [x] `Benchstone.BenchI.Array2.Test` * Detected by: https://github.com/dotnet/runtime/issues/71059 * Explained by: https://github.com/dotnet/runtime/issues/71059#issuecomment-1203196867 * This regression was included in .NET 7.0 Preview 7 report https://github.com/dotnet/runtime/issues/73866. * [x] `Benchstone.MDBenchI.MDMulMatrix.Test` * Detected by: https://github.com/dotnet/runtime/issues/72030 * Explained by: https://github.com/dotnet/runtime/issues/72030#issuecomment-1213512373 * This regression was included in .NET 7.0 Preview 7 report https://github.com/dotnet/runtime/issues/73866. * Note: We noticed arm64 improvements despite x64 regressions. * [x] `System.Collections.Concurrent.IsEmpty.Dictionary(Size: 512)`, `System.Collections.Concurrent.IsEmpty.Dictionary(Size: 512)` * Introduced by https://github.com/dotnet/runtime/pull/73488 - * Explained by: https://github.com/dotnet/runtime/pull/73488#issuecomment-1245922399 * This regression was included in .NET 7.0 RC1 report https://github.com/dotnet/runtime/issues/76320. * [x] `System.Collections.IndexerSet.Array(Size: 512)` * Detected by: https://github.com/dotnet/perf-autofiling-issues/issues/6746 * Added comment: https://github.com/dotnet/runtime/pull/71659#issuecomment-1278221030 * [x] `System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False))`, `System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))`, `System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False))`, `System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))` * Explained by: https://github.com/dotnet/runtime/pull/69878 * Added comment: https://github.com/dotnet/runtime/pull/69878#issuecomment-1281528243 * This regression was incldued in .NET 7.0 preview 7 report https://github.com/dotnet/runtime/pull/69878. * [x] `System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt")`, `System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf")` * Detected by: https://github.com/dotnet/runtime/issues/73391 * Explained by: https://github.com/dotnet/runtime/issues/73391#issuecomment-1205619431 * Note: Perf test bug, will be fixed in https://github.com/dotnet/performance/pull/2567. Already discussed in .NET 7.0 RC1 report https://github.com/dotnet/runtime/issues/76320. * [x] `System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0])`, `System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1])`, `System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 0, 0])`, `System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 1, 1])` * Explained by: https://github.com/dotnet/runtime/pull/60463#issuecomment-945828201 * For all other cases, the perf drastically improved. * [x] `System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits)`, `System.Numerics.Tests.Perf_BigInteger.Add(arguments: 65536,65536 bits)` * Regression was reported by the bot (https://github.com/dotnet/runtime/issues/60293), but not for this particular argument * Added comment: https://github.com/dotnet/runtime/issues/60293#issuecomment-1072434031 * Explained by: https://github.com/dotnet/runtime/issues/60293#issuecomment-1083543708 * This regression was included in .NET 7.0 Preview 2 report https://github.com/dotnet/runtime/issues/66848. * [x] `System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark`, `System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark` * Introduced by: https://github.com/dotnet/runtime/pull/68691 * [x] `System.Numerics.Tests.Perf_VectorConvert.Convert_double_long`: * Tracked by: https://github.com/dotnet/runtime/issues/65189 * Explained by: https://github.com/dotnet/runtime/issues/65189#issuecomment-1036388904 * This regression was included in .NET 7.0 preview 1 report https://github.com/dotnet/runtime/issues/65200 * Note: As per @tannergooding, this regression can be fixed. * [x] `System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes.DeriveBytes` * specific to Unix, Windows seems to be unaffected * Explained by: https://github.com/dotnet/runtime/issues/65719#issuecomment-1048339034 * This regression was included in .NET 7.0 preview 2 report https://github.com/dotnet/runtime/issues/66848 * [x] `System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: Compiled)`, `System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled)`, `System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: Compiled)`, `System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?m)^Sherlock Holmes|Sherlock Holmes$", Options: Compiled)`, `System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)` * Explained by: https://github.com/dotnet/runtime/issues/65187#issuecomment-1036287653 * Note: While we were looking at the results, we noticed a mono-specific regression, created issue https://github.com/dotnet/runtime/issues/77129. * [x] `System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: Compiled)`, `System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: None)` * Explained by: https://github.com/dotnet/runtime/issues/67915#issuecomment-1096979972 * This regression was included in .NET 7.0 Preview 4 report https://github.com/dotnet/runtime/issues/69396. ### Investigation In Progress * [ ] `Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty` * Detected by: https://github.com/dotnet/runtime/issues/77900 * Added comment: https://github.com/dotnet/runtime/issues/77900 * [ ] `FractalPerf.Launch.Test` * Detected by: https://github.com/dotnet/perf-autofiling-issues/issues/4414, but not solved * the group of benchmark regressions in the above issue were addressed in https://github.com/dotnet/runtime/issues/67594 * Explained by: https://github.com/dotnet/runtime/issues/67594#issuecomment-1089180924 * Added comment as this went unresolved since preview 4 report: https://github.com/dotnet/runtime/issues/67594#issuecomment-1278172793 * [ ] `PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod`, `System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1])`, `System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128])`, `System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024])`, `System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0])`, `System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024]`, `System.Text.Tests.Perf_Encoding.GetEncoder*` * Detected by: https://github.com/dotnet/runtime/issues/74014. * Explained by: https://github.com/dotnet/runtime/issues/74014#issuecomment-1230319972 * Added comments: https://github.com/dotnet/runtime/issues/74014#issuecomment-1278190959 * Note: GC related. * [ ] `System.Collections.Sort.Array_ComparerClass(Size: 512)`, `System.Collections.Sort.Array_ComparerStruct(Size: 512)`, `System.Collections.Sort.Array_Comparison(Size: 512)`, `System.Collections.Sort.LinqQuery(Size: 512)`. * Seems like noise on macOS and we thought that it would be good to ask for a re-run to the contributor. * [ ] `System.Buffers.Tests.ReadOnlySequenceTests.IterateGetPositionArray` * Created issue: https://github.com/dotnet/runtime/issues/77028 * Note: Gradual regression detected over .NET 7 release cycle. * [ ] `System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: List)` * Added comment: https://github.com/dotnet/runtime/pull/65738#issuecomment-1279259957 * [ ] `System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark` , `System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark`, `System.Numerics.Tests.Perf_Matrix4x4.Transpose`, `System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark`, `System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark` * Tracked here, added comment: https://github.com/dotnet/runtime/issues/65191#issuecomment-1279304302 * Already included in .NET 7.0 Preview 1 https://github.com/dotnet/runtime/issues/65200. * [ ] `System.Tests.Perf_GC.AllocateUninitializedArray(length: 10000, pinned: True)`, `System.Tests.Perf_GC.AllocateUninitializedArray(length: 10000, pinned: True)` * Added comment: https://github.com/dotnet/runtime/issues/65198#issuecomment-1279474550 * For non-macOS configurations, historical data indicates that this regression is solved already. We don't have historical data for Mac machines, so this is technically under investigation. * [ ] `System.Tests.Perf_String.CtorCharCount(size: 1)`, `System.Tests.Perf_String.CtorCharCount(size: 10)`, `System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)`, `System.Text.Tests.Perf_StringBuilder.ctor_capacity(length: 100000)`, `System.Tests.Perf_String.Trim(s: " Test")` * Added comment: https://github.com/dotnet/runtime/issues/77064#issuecomment-1279505432 * [ ] `System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)`, `System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)` * Detected by: https://github.com/dotnet/runtime/issues/68405 * Added comment: https://github.com/dotnet/runtime/issues/68405#issuecomment-1279535304 ### Noise, Flaky or Multimodal The following benchmarks showed up in the report generated by the tool, but were not actual regressions: * [x] `System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits)` * We are seeing 15-25% regressions in the report data but this doesn't match the historical data. * [x] `System.Numerics.Tests.Perf_Matrix4x4.LerpBenchmark` * [x] `System.Numerics.Tests.Perf_VectorOf<*>.*` ## Statistics Total: 86862 Same: 55.57 % Slower: 7.85 % Faster: 23.65 % Noise: 12.88 % Unknown: 0.04 % ## Statistics per Architecture | Architecture | Same | Slower | Faster | Noise | Unknown| | ------------ | ------- | ------- | ------- | ------- | ------- | | Arm64 | 50.16 % | 2.66 % | 34.81 % | 12.31 % | 0.05 % | | X64 | 58.07 % | 10.24 % | 18.51 % | 13.15 % | 0.04 % | ## Statistics per Operating System | Operating System | Same | Slower | Faster | Noise | Unknown| | ------------------- | ------- | ------- | ------- | ------- | ------- | | macOS Monterey 12.6 | 49.99 % | 2.60 % | 34.20 % | 13.16 % | 0.05 % | | ubuntu 18.04 | 61.50 % | 6.66 % | 21.50 % | 10.25 % | 0.09 % | | Windows 11 | 57.64 % | 6.26 % | 22.91 % | 13.15 % | 0.03 % | | debian 11 | 63.41 % | 7.07 % | 16.30 % | 13.17 % | 0.04 % | | macOS Big Sur 11.7 | 63.45 % | 3.35 % | 19.18 % | 13.94 % | 0.07 % | | ubuntu 20.04 | 49.65 % | 13.46 % | 22.58 % | 14.29 % | 0.01 % | | Windows 10 | 42.13 % | 26.62 % | 18.84 % | 12.42 % | 0.00 % | ## Statistics per Namespace

| Namespace | Same | Slower | Faster | Noise | Unknown| | --------------------------------------------------------- | ------- | ------- | ------- | ------- | ------- | | Benchmark | 56.58 % | 32.89 % | 10.53 % | 0.00 % | 0.00 % | | BenchmarksGame | 65.41 % | 8.52 % | 26.07 % | 0.00 % | 0.00 % | | Benchstone.BenchF | 76.84 % | 6.05 % | 17.11 % | 0.00 % | 0.00 % | | Benchstone.BenchI | 53.95 % | 10.09 % | 35.96 % | 0.00 % | 0.00 % | | Benchstone.MDBenchF | 42.11 % | 3.16 % | 54.74 % | 0.00 % | 0.00 % | | Benchstone.MDBenchI | 20.00 % | 9.47 % | 70.53 % | 0.00 % | 0.00 % | | Devirtualization | 65.26 % | 2.11 % | 1.05 % | 31.58 % | 0.00 % | | Exceptions | 77.05 % | 1.32 % | 21.64 % | 0.00 % | 0.00 % | | FractalPerf | 5.26 % | 89.47 % | 5.26 % | 0.00 % | 0.00 % | | GuardedDevirtualization | 69.42 % | 0.00 % | 4.51 % | 26.07 % | 0.00 % | | IfStatements | 33.68 % | 0.00 % | 66.32 % | 0.00 % | 0.00 % | | Inlining | 33.33 % | 0.00 % | 1.75 % | 64.91 % | 0.00 % | | Interop | 89.47 % | 0.00 % | 10.53 % | 0.00 % | 0.00 % | | JetStream | 34.21 % | 2.63 % | 63.16 % | 0.00 % | 0.00 % | | Layout | 84.21 % | 0.00 % | 15.79 % | 0.00 % | 0.00 % | | Lowering | 84.21 % | 0.00 % | 0.00 % | 15.79 % | 0.00 % | | MicroBenchmarks.Serializers | 62.27 % | 3.75 % | 33.98 % | 0.00 % | 0.00 % | | Microsoft.AspNetCore.Server.Kestrel.Performance | 63.16 % | 10.53 % | 26.32 % | 0.00 % | 0.00 % | | Microsoft.Extensions.Caching.Memory.Tests | 69.38 % | 13.88 % | 16.75 % | 0.00 % | 0.00 % | | Microsoft.Extensions.Configuration | 65.50 % | 25.15 % | 9.36 % | 0.00 % | 0.00 % | | Microsoft.Extensions.Configuration.Xml | 65.79 % | 5.26 % | 28.95 % | 0.00 % | 0.00 % | | Microsoft.Extensions.DependencyInjection | 45.67 % | 3.57 % | 48.90 % | 1.87 % | 0.00 % | | Microsoft.Extensions.Logging | 64.66 % | 3.16 % | 28.42 % | 3.76 % | 0.00 % | | Microsoft.Extensions.Primitives.Performance | 50.00 % | 0.66 % | 1.32 % | 48.03 % | 0.00 % | | Microsoft.Extensions.Primitives | 42.40 % | 0.29 % | 17.84 % | 39.47 % | 0.00 % | | PerfLabTests | 56.18 % | 12.70 % | 24.47 % | 6.64 % | 0.00 % | | PerfLabTests.CastingPerf2 | 49.12 % | 14.04 % | 36.84 % | 0.00 % | 0.00 % | | SciMark2 | 64.21 % | 1.05 % | 34.74 % | 0.00 % | 0.00 % | | SIMD | 90.06 % | 0.58 % | 9.36 % | 0.00 % | 0.00 % | | Span | 79.79 % | 0.42 % | 19.79 % | 0.00 % | 0.00 % | | StoreBlock | 52.81 % | 0.00 % | 4.04 % | 43.16 % | 0.00 % | | System.Buffers.Binary.Tests | 43.42 % | 4.61 % | 25.66 % | 26.32 % | 0.00 % | | System.Buffers.Tests | 54.36 % | 10.93 % | 28.04 % | 6.67 % | 0.00 % | | System.Buffers.Text.Tests | 48.22 % | 3.79 % | 21.52 % | 26.47 % | 0.00 % | | System.Collections | 63.24 % | 7.83 % | 27.74 % | 1.19 % | 0.00 % | | System.Collections.Concurrent | 67.46 % | 7.75 % | 12.96 % | 11.83 % | 0.00 % | | System.Collections.Tests | 53.76 % | 5.46 % | 28.87 % | 11.90 % | 0.00 % | | System.ComponentModel.Tests | 38.76 % | 0.48 % | 60.77 % | 0.00 % | 0.00 % | | System.ConsoleTests | 69.30 % | 8.77 % | 21.93 % | 0.00 % | 0.00 % | | System.Diagnostics | 49.47 % | 7.11 % | 43.42 % | 0.00 % | 0.00 % | | System.Diagnostics.Tracing | 75.00 % | 9.21 % | 15.79 % | 0.00 % | 0.00 % | | System.Formats.Cbor.Tests | 65.99 % | 4.05 % | 29.96 % | 0.00 % | 0.00 % | | System.Globalization.Tests | 75.37 % | 6.03 % | 15.79 % | 2.81 % | 0.00 % | | System | 65.20 % | 3.51 % | 28.07 % | 3.22 % | 0.00 % | | System.IO.Compression | 80.92 % | 3.29 % | 15.79 % | 0.00 % | 0.00 % | | System.IO.MemoryMappedFiles.Tests | 26.36 % | 4.09 % | 69.55 % | 0.00 % | 0.00 % | | System.IO.Pipelines.Tests | 69.74 % | 1.32 % | 28.95 % | 0.00 % | 0.00 % | | System.IO.Pipes.Tests | 73.68 % | 16.67 % | 9.65 % | 0.00 % | 0.00 % | | System.IO.Tests | 60.55 % | 12.12 % | 24.61 % | 2.72 % | 0.00 % | | System.Linq.Tests | 77.64 % | 7.45 % | 14.17 % | 0.73 % | 0.00 % | | System.MathBenchmarks | 87.83 % | 1.81 % | 8.47 % | 1.89 % | 0.00 % | | System.Memory | 51.32 % | 1.89 % | 19.58 % | 27.21 % | 0.00 % | | System.Net.Http.Tests | 42.11 % | 31.58 % | 26.32 % | 0.00 % | 0.00 % | | System.Net.NetworkInformation.Tests | 42.11 % | 19.74 % | 38.16 % | 0.00 % | 0.00 % | | System.Net.Primitives.Tests | 60.29 % | 9.57 % | 15.55 % | 14.59 % | 0.00 % | | System.Net.Security.Tests | 74.10 % | 14.43 % | 11.48 % | 0.00 % | 0.00 % | | System.Net.Sockets.Tests | 49.45 % | 23.08 % | 27.47 % | 0.00 % | 0.00 % | | System.Net.Tests | 71.56 % | 13.76 % | 14.68 % | 0.00 % | 0.00 % | | System.Net.WebSockets.Tests | 47.37 % | 39.47 % | 13.16 % | 0.00 % | 0.00 % | | System.Numerics.Tests | 44.69 % | 5.24 % | 6.04 % | 44.03 % | 0.00 % | | System.Reflection | 30.50 % | 2.15 % | 64.35 % | 2.99 % | 0.00 % | | System.Runtime.InteropServices.Tests | 78.95 % | 2.63 % | 15.79 % | 2.63 % | 0.00 % | | System.Runtime.Serialization.Formatters.Tests | 42.11 % | 52.63 % | 5.26 % | 0.00 % | 0.00 % | | System.Security.Cryptography.Primitives.Tests.Performance | 84.87 % | 9.21 % | 5.92 % | 0.00 % | 0.00 % | | System.Security.Cryptography.Tests | 69.59 % | 7.02 % | 23.39 % | 0.00 % | 0.00 % | | System.Security.Cryptography.X509Certificates.Tests | 73.68 % | 12.28 % | 14.04 % | 0.00 % | 0.00 % | | System.Tests | 50.35 % | 12.24 % | 25.46 % | 11.95 % | 0.00 % | | System.Text.Encodings.Web.Tests | 72.59 % | 14.47 % | 11.84 % | 1.10 % | 0.00 % | | System.Text.Json.Document.Tests | 60.21 % | 10.53 % | 29.26 % | 0.00 % | 0.00 % | | System.Text.Json.Reader.Tests | 46.05 % | 15.79 % | 38.16 % | 0.00 % | 0.00 % | | System.Text.Json.Serialization.Tests | 57.60 % | 11.37 % | 31.03 % | 0.00 % | 0.00 % | | System.Text.Json.Tests | 67.98 % | 11.07 % | 20.59 % | 0.35 % | 0.00 % | | System.Text | 63.51 % | 24.21 % | 12.28 % | 0.00 % | 0.00 % | | System.Text.RegularExpressions.Tests | 16.73 % | 5.44 % | 76.91 % | 0.00 % | 0.92 % | | System.Text.Tests | 57.52 % | 21.93 % | 18.67 % | 1.88 % | 0.00 % | | System.Threading.Channels.Tests | 54.39 % | 17.54 % | 28.07 % | 0.00 % | 0.00 % | | System.Threading.Tasks.Tests | 47.37 % | 19.30 % | 33.33 % | 0.00 % | 0.00 % | | System.Threading.Tasks | 69.55 % | 17.67 % | 11.65 % | 1.13 % | 0.00 % | | System.Threading.Tests | 58.90 % | 11.31 % | 17.13 % | 12.65 % | 0.00 % | | System.Xml.Linq | 59.00 % | 9.70 % | 22.16 % | 9.14 % | 0.00 % | | System.Xml.Tests | 5.26 % | 2.11 % | 92.63 % | 0.00 % | 0.00 % | | V8.Crypto | 78.95 % | 15.79 % | 5.26 % | 0.00 % | 0.00 % | | V8.Richards | 73.68 % | 5.26 % | 21.05 % | 0.00 % | 0.00 % | | XmlDocumentTests.XmlDocumentTests | 40.35 % | 22.81 % | 21.05 % | 15.79 % | 0.00 % | | XmlDocumentTests.XmlNodeListTests | 86.84 % | 5.26 % | 2.63 % | 5.26 % | 0.00 % | | XmlDocumentTests.XmlNodeTests | 36.84 % | 2.63 % | 0.00 % | 60.53 % | 0.00 % | | System.Drawing.Tests | 77.50 % | 1.87 % | 15.00 % | 5.63 % | 0.00 % | | HardwareIntrinsics.RayTracer | 76.92 % | 7.69 % | 15.38 % | 0.00 % | 0.00 % |

**Big thanks to everyone involved!**

Author:	Jozkee
Assignees:	-
Labels:	`area-Meta`, `tenet-performance`, `tenet-performance-benchmarks`, `tracking`
Milestone:	Future

jjxtra commented 1 year ago

Hoping BigInteger performance improves. C# getting smoked by python computing pi digits is no bueno. https://programming-language-benchmarks.vercel.app/problem/pidigits

jeffhandley commented 1 year ago

Thank you, @jjxtra -- it's helpful to know that BigInteger performance is important to you. We've been accumulating issues around BigInteger and we've been thinking about what it might look like to reimplement it for better performance and reliability. That effort isn't going to be committed for .NET 8, but it is on our radar for the future.

It would be helpful data if you could add a comment to #41495 describing your scenarios and how BigInteger's performance affects you. Thanks again!

jjxtra commented 1 year ago

To be honest, it's probably not impacting anyone severely outside of either:

1] Benchmarks, which give C# a sour taste 2] Scientific calculations, which I don't have a use case for

As I mentioned above, someone seeing a benchmark where python beats C# would be puzzling

tannergooding commented 1 year ago

It's worth noting that just looking at "raw numbers" is also often not sufficient. You often have to compare the actual implementations, considering where devs have diverged algorithms or done customization as well.

In the case of using BigInteger to compute digits of PI, that's really the "naive" approach and isn't necessarily representative of a "real world use-case" for BigInteger. If you wanted to compute digits of PI fast, you'd likely use one of the "spigot" algorithms that allow computing digits without knowing or maintaining the state of all prior digits. This allows you to more easily parallelize and optimize the algorithm, all while taking up less memory/data.

There are certainly cases where we could/should improve the performance of BigInteger, but I'd like to try and drive those off of scenarios that matter the most and which are representative of practical use-cases.

danmoseley commented 1 year ago

I don't know about that site, but the CLBG which it is based on has two categories -- one for the big integer abstractions, and one for anything goes. On the latter, C# is faster than most. https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/pidigits.html#intrinsics

danmoseley commented 1 year ago

As always its great when community members help improve our submissions in both sites..

jjxtra commented 1 year ago

I don't know about that site, but the CLBG which it is based on has two categories -- one for the big integer abstractions, and one for anything goes. On the latter, C# is faster than most. https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/pidigits.html#intrinsics

Very interesting, thanks for the link.

danmoseley commented 1 year ago

Now I look at the impls there they are all using the GNU arbitrary precision library, so not too interesting perhaps from a .NET point of view. Anyway -- if this stuff interests you perhaps you'd like to look further at what could be done to make things faster, assuming it didn't impact other scenarios.

dotnet / runtime