dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

Regressions in System.Linq.Tests.Perf_Enumerable #97692

Closed performanceautofiler[bot] closed 8 months ago

performanceautofiler[bot] commented 9 months ago

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 1a985d9d113b3873f337cead6c3a136e1876cd2c
Compare 72c4c57a3c8c1b267447f253fdc14f9d14d91a2b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
301.14 ns 323.99 ns 1.08 0.01 False
10.37 ns 28.45 ns 2.74 0.27 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'
### Payloads [Baseline]() [Compare]() ### System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: List) #### ETL Files #### Histogram #### JIT Disasms ### System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
cincuranet commented 9 months ago

Only EmptyTakeSelectToArray seems to be valid. Diff.

Regression is on all platforms: image

ghost commented 9 months ago

Tagging subscribers to this area: @dotnet/area-system-linq See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | arm64 OS | Windows 10.0.19041 Queue | SurfaceWindows Baseline | [1a985d9d113b3873f337cead6c3a136e1876cd2c](https://github.com/dotnet/runtime/commit/1a985d9d113b3873f337cead6c3a136e1876cd2c) Compare | [72c4c57a3c8c1b267447f253fdc14f9d14d91a2b](https://github.com/dotnet/runtime/commit/72c4c57a3c8c1b267447f253fdc14f9d14d91a2b) Diff | [Diff](https://github.com/dotnet/runtime/compare/1a985d9d113b3873f337cead6c3a136e1876cd2c...72c4c57a3c8c1b267447f253fdc14f9d14d91a2b) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Linq.Tests.Perf_Enumerable Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio -- | -- | -- | -- | -- | -- | -- | -- | -- |
  • [WhereAny_LastElementMatches - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [📈 - ADX Test Multi Config Graph]()
| 301.14 ns | 323.99 ns | 1.08 | 0.01 | False | | | |
  • [EmptyTakeSelectToArray - Duration of single invocation]()
  • 📝 - [Benchmark Source]()
  • [📈 - ADX Test Multi Config Graph]()
| 10.37 ns | 28.45 ns | 2.74 | 0.27 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*' ```
### Payloads [Baseline]() [Compare]() ### System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: List) #### ETL Files #### Histogram #### JIT Disasms ### System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray #### ETL Files #### Histogram #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `arch-arm64`, `area-System.Linq`, `os-windows`, `untriaged`, `runtime-coreclr`
Milestone: -
stephentoub commented 8 months ago

Only EmptyTakeSelectToArray seems to be valid. Diff.

I think we need to accept this one. The gap has come back down a bit, to less than 10ns. The issue is we're now checking for empty arrays rather than a dedicated empty sentinel, and the perf test is using T == int, which requires covariance checks (since eg a uint[] will also pass as an int[]). So the couple of times on this path that we're testing for empty, that check is a bit more expensive than it used to be. However, it's also more comprehensive than it used to be, so whereas previously we'd only have special-cased empty inputs that came from Enumerable.Empty<T>, now we'll special-case both that and inputs from Array.Empty<T>() (which is what collection literals produce) and new T[i] where i == 0, etc.