search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/arm64: 6 Improvements on 10/6/2023 11:50:29 PM #23011

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline f55549222e4dc1469190caa56e03361ee46208ca
Compare f21a2666c577306e437f80fe934d76cdb15072a5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.56 ms 1.47 ms 0.94 0.00 False
320.86 μs 303.46 μs 0.95 0.00 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson", Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 8/27/2023 6:19:41 AM, 10/6/2023 9:14:38 AM, 10/11/2023 7:15:38 PM falls between 10/2/2023 7:07:56 PM and 10/11/2023 7:15:38 PM. IsImprovementStdDev: Marked as improvement because 19.09759874396002 (T) = (0 -1461489.06503219) / Math.Sqrt((479213264.23872846 / (299)) + (314736819.999678 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.08765482028752052 = (1601903.6407829437 - 1461489.06503219) / 1601903.6407829437 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/15/2023 5:11:34 PM, 10/6/2023 9:14:38 AM, 10/11/2023 7:15:38 PM falls between 10/2/2023 7:07:56 PM and 10/11/2023 7:15:38 PM. IsImprovementStdDev: Marked as improvement because 34.34964125501859 (T) = (0 -302916.3169496102) / Math.Sqrt((18585399.119986266 / (299)) + (1849959.3200064418 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.06456518894950103 = (323824.0798516289 - 302916.3169496102) / 323824.0798516289 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline f55549222e4dc1469190caa56e03361ee46208ca
Compare f21a2666c577306e437f80fe934d76cdb15072a5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in ByteMark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
834.98 ms 765.76 ms 0.92 0.00 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'ByteMark*'
### Payloads [Baseline]() [Compare]() ### ByteMark.BenchNeural #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 9/28/2023 11:34:37 PM, 10/6/2023 9:14:38 AM, 10/11/2023 7:15:38 PM falls between 10/2/2023 7:07:56 PM and 10/11/2023 7:15:38 PM. IsImprovementStdDev: Marked as improvement because 196.09108890703018 (T) = (0 -766955150.3601953) / Math.Sqrt((40055164876125.29 / (299)) + (690788109546.3832 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.11316527525904561 = (864823093.8230615 - 766955150.3601953) / 864823093.8230615 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline f55549222e4dc1469190caa56e03361ee46208ca
Compare f21a2666c577306e437f80fe934d76cdb15072a5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.68 ms 2.53 ms 0.94 0.03 False
42.89 ms 39.55 ms 0.92 0.01 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'
### Payloads [Baseline]() [Compare]() ### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\\p{Sm}", Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/9/2023 7:27:03 AM, 5/9/2023 11:02:55 PM, 10/6/2023 9:14:38 AM, 10/11/2023 7:15:38 PM falls between 10/2/2023 7:07:56 PM and 10/11/2023 7:15:38 PM. IsImprovementStdDev: Marked as improvement because 29.52349312640571 (T) = (0 -2537453.543792517) / Math.Sqrt((8078663426.1844225 / (299)) + (10915076.490233384 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.058807354506667145 = (2695998.0572972842 - 2537453.543792517) / 2695998.0572972842 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: Compiled) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 10/6/2023 9:14:38 AM, 10/11/2023 7:15:38 PM falls between 10/2/2023 7:07:56 PM and 10/11/2023 7:15:38 PM. IsImprovementStdDev: Marked as improvement because 90.86402005454875 (T) = (0 -39326837.91819292) / Math.Sqrt((244180679550.03146 / (299)) + (13051347210.77212 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.11219941669014354 = (44296927.32525186 - 39326837.91819292) / 44296927.32525186 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline f55549222e4dc1469190caa56e03361ee46208ca
Compare f21a2666c577306e437f80fe934d76cdb15072a5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in Benchstone.MDBenchF.MDLLoops

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
961.26 ms 909.32 ms 0.95 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchF.MDLLoops*'
### Payloads [Baseline]() [Compare]() ### Benchstone.MDBenchF.MDLLoops.Test #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 9/28/2023 11:34:37 PM, 10/6/2023 9:14:38 AM, 10/11/2023 7:15:38 PM falls between 10/2/2023 7:07:56 PM and 10/11/2023 7:15:38 PM. IsImprovementStdDev: Marked as improvement because 102.02913900055998 (T) = (0 -909138560.8058606) / Math.Sqrt((206502588186699.5 / (299)) + (60789564582.01882 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.0858790665109412 = (994549547.548177 - 909138560.8058606) / 994549547.548177 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 1 year ago

https://github.com/dotnet/runtime/pull/92710