search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.19k stars 4.72k forks source link

[Perf] Regressions from flow graph reordering change #70159

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IDictionary - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.ContainsKeyTrue(Int32%2c%20Int32).IDictionary(Size%3a%20512).html>) 6.16 μs 6.74 μs 1.09 0.02 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.737065667380443 > 6.456760085057447. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -41.18520937234114 (T) = (0 -6702.308505913001) / Math.Sqrt((1380.3406537166666 / (29)) + (1275.045897158201 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.09234589955871175 = (6135.701620357264 - 6702.308505913001) / 6135.701620357264 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Memory.Constructors<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadOnlySpanFromArray - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Memory.Constructors(Byte).ReadOnlySpanFromArray.html>) 0.11 ns 1.14 ns 10.40 0.69 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Constructors&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Constructors<Byte>.ReadOnlySpanFromArray ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1434632033857692 > 0.1891852574345431. IsChangePoint: Marked as a change because one of 2/1/2022 4:18:38 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -19.990188823533767 (T) = (0 -0.9257845523005314) / Math.Sqrt((0.002826405902611865 / (29)) + (0.011484976696724828 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -4.006572236919883 = (0.18491385093248702 - 0.9257845523005314) / 0.18491385093248702 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableArray - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.IterateForEach(String).ImmutableArray(Size%3a%20512).html>) 457.25 ns 498.06 ns 1.09 0.00 True

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IterateForEach<String>.ImmutableArray(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 498.0561399282693 > 480.0857621193237. IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -596.4214656864929 (T) = (0 -498.1333366046504) / Math.Sqrt((0.11315161591925187 / (29)) + (0.005949614232756872 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.08799380264675961 = (457.8457481952956 - 498.1333366046504) / 457.8457481952956 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Queue - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.Concurrent.IsEmpty(Int32).Queue(Size%3a%20512).html>) 11.28 ns 14.68 ns 1.30 0.21 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.IsEmpty&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Concurrent.IsEmpty<Int32>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.683828767887277 > 11.919290558200084. IsChangePoint: Marked as a change because one of 2/3/2022 8:57:20 PM, 3/15/2022 12:53:39 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -11.07923240330845 (T) = (0 -14.93100806697449) / Math.Sqrt((0.3966814480625782 / (29)) + (0.6743232996875372 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.28349057113305665 = (11.633126415407554 - 14.93100806697449) / 11.633126415407554 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IDictionary - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.TryGetValueTrue(Int32%2c%20Int32).IDictionary(Size%3a%20512).html>) 6.24 μs 7.09 μs 1.14 0.02 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.094232775331843 > 6.614450639204545. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -48.80556660224299 (T) = (0 -7130.090187117306) / Math.Sqrt((4198.404554838031 / (29)) + (1019.9229245797691 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.12355503582863395 = (6346.008837794749 - 7130.090187117306) / 6346.008837794749 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

category:performance theme:benchmarks

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

DrewScoggins commented 2 years ago

Seems related to https://github.com/dotnet/runtime/pull/69878

DrewScoggins commented 2 years ago

All non-IndexOf tests related here. https://github.com/dotnet/perf-autofiling-issues/issues/5781

ghost commented 2 years ago

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [486b4d1a36aef5bbe6a77bb2c3412772e712961e](https://github.com/dotnet/runtime/commit/486b4d1a36aef5bbe6a77bb2c3412772e712961e) Compare | [afacf6a99e1b98331af958caec638bd93dc46ffe](https://github.com/dotnet/runtime/commit/afacf6a99e1b98331af958caec638bd93dc46ffe) Diff | [Diff](https://github.com/dotnet/runtime/compare/486b4d1a36aef5bbe6a77bb2c3412772e712961e...afacf6a99e1b98331af958caec638bd93dc46ffe) ### Regressions in System.Collections.ContainsKeyTrue<Int32, Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IDictionary - Duration of single invocation]() | 6.16 μs | 6.74 μs | 1.09 | 0.02 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue<Int32, Int32>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.737065667380443 > 6.456760085057447. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -41.18520937234114 (T) = (0 -6702.308505913001) / Math.Sqrt((1380.3406537166666 / (29)) + (1275.045897158201 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.09234589955871175 = (6135.701620357264 - 6702.308505913001) / 6135.701620357264 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [486b4d1a36aef5bbe6a77bb2c3412772e712961e](https://github.com/dotnet/runtime/commit/486b4d1a36aef5bbe6a77bb2c3412772e712961e) Compare | [afacf6a99e1b98331af958caec638bd93dc46ffe](https://github.com/dotnet/runtime/commit/afacf6a99e1b98331af958caec638bd93dc46ffe) Diff | [Diff](https://github.com/dotnet/runtime/compare/486b4d1a36aef5bbe6a77bb2c3412772e712961e...afacf6a99e1b98331af958caec638bd93dc46ffe) ### Regressions in System.Memory.Constructors<Byte> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ReadOnlySpanFromArray - Duration of single invocation]() | 0.11 ns | 1.14 ns | 10.40 | 0.69 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Constructors<Byte>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Constructors<Byte>.ReadOnlySpanFromArray ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1434632033857692 > 0.1891852574345431. IsChangePoint: Marked as a change because one of 2/1/2022 4:18:38 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -19.990188823533767 (T) = (0 -0.9257845523005314) / Math.Sqrt((0.002826405902611865 / (29)) + (0.011484976696724828 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -4.006572236919883 = (0.18491385093248702 - 0.9257845523005314) / 0.18491385093248702 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [486b4d1a36aef5bbe6a77bb2c3412772e712961e](https://github.com/dotnet/runtime/commit/486b4d1a36aef5bbe6a77bb2c3412772e712961e) Compare | [afacf6a99e1b98331af958caec638bd93dc46ffe](https://github.com/dotnet/runtime/commit/afacf6a99e1b98331af958caec638bd93dc46ffe) Diff | [Diff](https://github.com/dotnet/runtime/compare/486b4d1a36aef5bbe6a77bb2c3412772e712961e...afacf6a99e1b98331af958caec638bd93dc46ffe) ### Regressions in System.Collections.IterateForEach<String> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ImmutableArray - Duration of single invocation]() | 457.25 ns | 498.06 ns | 1.09 | 0.00 | True | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach<String>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IterateForEach<String>.ImmutableArray(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 498.0561399282693 > 480.0857621193237. IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -596.4214656864929 (T) = (0 -498.1333366046504) / Math.Sqrt((0.11315161591925187 / (29)) + (0.005949614232756872 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.08799380264675961 = (457.8457481952956 - 498.1333366046504) / 457.8457481952956 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [486b4d1a36aef5bbe6a77bb2c3412772e712961e](https://github.com/dotnet/runtime/commit/486b4d1a36aef5bbe6a77bb2c3412772e712961e) Compare | [afacf6a99e1b98331af958caec638bd93dc46ffe](https://github.com/dotnet/runtime/commit/afacf6a99e1b98331af958caec638bd93dc46ffe) Diff | [Diff](https://github.com/dotnet/runtime/compare/486b4d1a36aef5bbe6a77bb2c3412772e712961e...afacf6a99e1b98331af958caec638bd93dc46ffe) ### Regressions in System.Collections.Concurrent.IsEmpty<Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Queue - Duration of single invocation]() | 11.28 ns | 14.68 ns | 1.30 | 0.21 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.IsEmpty<Int32>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Concurrent.IsEmpty<Int32>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.683828767887277 > 11.919290558200084. IsChangePoint: Marked as a change because one of 2/3/2022 8:57:20 PM, 3/15/2022 12:53:39 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -11.07923240330845 (T) = (0 -14.93100806697449) / Math.Sqrt((0.3966814480625782 / (29)) + (0.6743232996875372 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.28349057113305665 = (11.633126415407554 - 14.93100806697449) / 11.633126415407554 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [486b4d1a36aef5bbe6a77bb2c3412772e712961e](https://github.com/dotnet/runtime/commit/486b4d1a36aef5bbe6a77bb2c3412772e712961e) Compare | [afacf6a99e1b98331af958caec638bd93dc46ffe](https://github.com/dotnet/runtime/commit/afacf6a99e1b98331af958caec638bd93dc46ffe) Diff | [Diff](https://github.com/dotnet/runtime/compare/486b4d1a36aef5bbe6a77bb2c3412772e712961e...afacf6a99e1b98331af958caec638bd93dc46ffe) ### Regressions in System.Collections.TryGetValueTrue<Int32, Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IDictionary - Duration of single invocation]() | 6.24 μs | 7.09 μs | 1.14 | 0.02 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryGetValueTrue<Int32, Int32>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.094232775331843 > 6.614450639204545. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM. IsRegressionStdDev: Marked as regression because -48.80556660224299 (T) = (0 -7130.090187117306) / Math.Sqrt((4198.404554838031 / (29)) + (1019.9229245797691 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.12355503582863395 = (6346.008837794749 - 7130.090187117306) / 6346.008837794749 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: AndyAyersMS
Labels: `tenet-performance`, `tenet-performance-benchmarks`, `area-CodeGen-coreclr`, `untriaged`, `refs/heads/main`, `RunKind=micro`, `Windows 10.0.19041`, `Regression`, `CoreClr`, `arm64`
Milestone: -
AndyAyersMS commented 2 years ago

We are not going to be able to address these in 7.0, so moving to 8.0.

AndyAyersMS commented 1 year ago

System.Collections.IterateForEach.ImmutableArray(Size: 512)

is since fixed

newplot - 2023-04-22T082420 231

Nov 2022 was https://github.com/dotnet/runtime/pull/77238 Jan 2023 was probably https://github.com/dotnet/runtime/pull/79346

AndyAyersMS commented 1 year ago

Performance of the other tests here are now dramatically improved with Dynamic PGO enabled by default newplot - 2023-06-01T153429 144

newplot - 2023-06-01T153741 084