search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.21k stars 4.72k forks source link

[Perf] Linux/x64: Regressions in System.Text.Tests.Perf_StringBuilder #107042

Closed performanceautofiler[bot] closed 1 month ago

performanceautofiler[bot] commented 2 months ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline bb1b43a6a155eaeb45953a3c87f6e04b12979778
Compare d3da7a2c823245534e557f97013136d06a98e731
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
116.54 μs 123.20 μs 1.06 0.00 False
96.58 μs 102.18 μs 1.06 0.01 False
95.03 μs 100.67 μs 1.06 0.01 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'
### System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100000) #### ETL Files #### Histogram #### JIT Disasms ### System.Text.Tests.Perf_StringBuilder.ctor_string(length: 100000) #### ETL Files #### Histogram #### JIT Disasms ### System.Text.Tests.Perf_StringBuilder.ToString_SingleSegment(length: 100000) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline bb1b43a6a155eaeb45953a3c87f6e04b12979778
Compare d3da7a2c823245534e557f97013136d06a98e731
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
19.66 ns 38.79 ns 1.97 0.02 False
96.11 ns 135.16 ns 1.41 0.03 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'
### System.Tests.Perf_String.IndexerCheckBoundCheckHoist #### ETL Files #### Histogram #### JIT Disasms ### System.Tests.Perf_String.IndexerCheckPathLength #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline bb1b43a6a155eaeb45953a3c87f6e04b12979778
Compare d3da7a2c823245534e557f97013136d06a98e731
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.12 μs 6.32 μs 1.23 0.06 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'
### System.Collections.ContainsTrue<Int32>.SortedSet(Size: 512) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline bb1b43a6a155eaeb45953a3c87f6e04b12979778
Compare d3da7a2c823245534e557f97013136d06a98e731
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrueComparer<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.11 μs 6.47 μs 1.27 0.07 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer&lt;Int32&gt;*'
### System.Collections.ContainsTrueComparer<Int32>.SortedSet(Size: 512) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline bb1b43a6a155eaeb45953a3c87f6e04b12979778
Compare d3da7a2c823245534e557f97013136d06a98e731
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.BenchI.BenchE

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
225.99 ms 291.57 ms 1.29 0.03 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.BenchE*'
### Benchstone.BenchI.BenchE.Test #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
LoopedBard3 commented 2 months ago

Only PR in the compare list: https://github.com/dotnet/runtime/pull/106578 @jkotas. It is not clear how this PR would cause all of this impact though (Regressions and Improvements). Only the Regressions in System.Text.Tests.Perf_StringBuilder seem likely to be not noise, keeping the rest just incase though.

LoopedBard3 commented 2 months ago

Other regressions: Linux x64: https://github.com/dotnet/perf-autofiling-issues/issues/40266, https://github.com/dotnet/perf-autofiling-issues/issues/40246, https://github.com/dotnet/perf-autofiling-issues/issues/40247

jkotas commented 2 months ago

Only PR in the compare list: https://github.com/dotnet/runtime/pull/106578 @jkotas. It is not clear how this PR would cause all of this impact though

We have picked up build images with new version of the C/C++ compilers at the same time. The build image updates are not submitted as PRs. They just happen in the background.

This PR fixed build break introduced by the new version of the compilers, so the regression got attributed to it. I believe that the C/C++ compiler update is responsible for the regressions and improvements. I expect that some of these will go away once we pick up fresh C/C++ PGO data collected on binaries produced by the new compilers.

dotnet-policy-service[bot] commented 2 months ago

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.