search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

[Perf] Linux/arm64: 3 Regressions on 2/27/2024 9:28:58 PM #99124

Closed performanceautofiler[bot] closed 6 months ago

performanceautofiler[bot] commented 8 months ago

Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline 463869be4569c9557cb1545c5e5850fecd5f9cd0
Compare e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
629.34 μs 702.45 μs 1.12 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsFalse<String>.LinkedList(Size: 512) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline 463869be4569c9557cb1545c5e5850fecd5f9cd0
Compare e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_Dictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
6.03 ms 7.53 ms 1.25 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_Dictionary*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Tests.Perf_Dictionary.ContainsValue(Items: 3000) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline 463869be4569c9557cb1545c5e5850fecd5f9cd0
Compare e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
162.01 ns 172.25 ns 1.06 0.04 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 8 months ago

https://github.com/dotnet/runtime/pull/98324 cc @AndyAyersMS

ghost commented 8 months ago

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | arm64 OS | ubuntu 22.04 Queue | AmpereUbuntu Baseline | [463869be4569c9557cb1545c5e5850fecd5f9cd0](https://github.com/dotnet/runtime/commit/463869be4569c9557cb1545c5e5850fecd5f9cd0) Compare | [e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073](https://github.com/dotnet/runtime/commit/e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073) Diff | [Diff](https://github.com/dotnet/runtime/compare/463869be4569c9557cb1545c5e5850fecd5f9cd0...e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Collections.ContainsFalse<String> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [LinkedList - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [📈 - ADX Test Multi Config Graph]()
| 629.34 μs | 702.45 μs | 1.12 | 0.01 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<String>*' ```
### Payloads [Baseline]() [Compare]() ### System.Collections.ContainsFalse<String>.LinkedList(Size: 512) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | arm64 OS | ubuntu 22.04 Queue | AmpereUbuntu Baseline | [463869be4569c9557cb1545c5e5850fecd5f9cd0](https://github.com/dotnet/runtime/commit/463869be4569c9557cb1545c5e5850fecd5f9cd0) Compare | [e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073](https://github.com/dotnet/runtime/commit/e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073) Diff | [Diff](https://github.com/dotnet/runtime/compare/463869be4569c9557cb1545c5e5850fecd5f9cd0...e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Collections.Tests.Perf_Dictionary Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [ContainsValue - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [📈 - ADX Test Multi Config Graph]()
| 6.03 ms | 7.53 ms | 1.25 | 0.01 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_Dictionary*' ```
### Payloads [Baseline]() [Compare]() ### System.Collections.Tests.Perf_Dictionary.ContainsValue(Items: 3000) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | arm64 OS | ubuntu 22.04 Queue | AmpereUbuntu Baseline | [463869be4569c9557cb1545c5e5850fecd5f9cd0](https://github.com/dotnet/runtime/commit/463869be4569c9557cb1545c5e5850fecd5f9cd0) Compare | [e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073](https://github.com/dotnet/runtime/commit/e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073) Diff | [Diff](https://github.com/dotnet/runtime/compare/463869be4569c9557cb1545c5e5850fecd5f9cd0...e101ae2bd1c198ba7aaa209d1a4c55d6ce6b4073) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [Email_IsMatch - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [📈 - ADX Test Multi Config Graph]()
| 162.01 ns | 172.25 ns | 1.06 | 0.04 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' ```
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: IgnoreCase, Compiled) #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `arch-arm64`, `os-linux`, `tenet-performance`, `tenet-performance-benchmarks`, `area-CodeGen-coreclr`, `runtime-coreclr`
Milestone: -
AndyAyersMS commented 6 months ago

newplot - 2024-04-16T125624 131 newplot - 2024-04-16T125612 655 newplot - 2024-04-16T125607 755

Two of the three improved, the other is now bimodal.

AndyAyersMS commented 6 months ago

Also System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count on arm64 surface is still a bit slower, but no other platform has regressed.

image

Going to close this.