search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.12k stars 4.7k forks source link

[Perf] Regressions in System.Globalization.Tests.StringSearch #76885

Open DrewScoggins opened 2 years ago

DrewScoggins commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline eb02b202c281188e9187d3c292cc223107d35a25
Compare 6dc320ed049427c93a6d04c51494d1072c376d62
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(%2c%20IgnoreCase%2c%20False)).html>) 206.66 ns 229.92 ns 1.11 0.01 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(en-US%2c%20None%2c%20False)).html>) 157.66 ns 181.28 ns 1.15 0.01 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(en-US%2c%20IgnoreNonSpace%2c%20False)).html>) 157.44 ns 181.38 ns 1.15 0.01 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(%2c%20None%2c%20False)).html>) 157.45 ns 181.39 ns 1.15 0.01 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(en-US%2c%20IgnoreCase%2c%20False)).html>) 206.70 ns 229.91 ns 1.11 0.01 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 229.92249249794838 > 216.9886323646198. IsChangePoint: Marked as a change because one of 8/15/2022 5:23:15 PM, 8/17/2022 6:09:21 PM, 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -511.1758886379049 (T) = (0 -229.91438151755705) / Math.Sqrt((0.09403051199595515 / (46)) + (0.00013157600581682603 / (2))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (2) - 2, .025) and -0.11374256555709585 = (206.43404376176778 - 229.91438151755705) / 206.43404376176778 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.2754494076922 > 165.27447788978907. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -587.1745781288312 (T) = (0 -181.2439522304857) / Math.Sqrt((0.03207028389040226 / (49)) + (0.0019841443439575675 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1513574089450762 = (157.41762794278563 - 181.2439522304857) / 157.41762794278563 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.37783263034405 > 165.30027593373453. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -206.95138528770087 (T) = (0 -181.49162825910662) / Math.Sqrt((0.02835066301893864 / (49)) + (0.025898890250939163 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1529053477261196 = (157.42110019444647 - 181.49162825910662) / 157.42110019444647 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.38782301127938 > 165.30941049594205. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -1069.165213210168 (T) = (0 -181.38718382011413) / Math.Sqrt((0.024614597945961592 / (49)) + (8.171306914636648E-07 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1522914146690745 = (157.41433244315766 - 181.38718382011413) / 157.41433244315766 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 229.9117129107842 > 216.87844739024908. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -197.4211601971299 (T) = (0 -230.00988660654446) / Math.Sqrt((0.23150453470920937 / (50)) + (0.01927614907845868 / (2))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (2) - 2, .025) and -0.11423760004232615 = (206.42804245504473 - 230.00988660654446) / 206.42804245504473 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

DrewScoggins commented 2 years ago

Seems related to https://github.com/dotnet/runtime/pull/73475

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-memory See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [eb02b202c281188e9187d3c292cc223107d35a25](https://github.com/dotnet/runtime/commit/eb02b202c281188e9187d3c292cc223107d35a25) Compare | [6dc320ed049427c93a6d04c51494d1072c376d62](https://github.com/dotnet/runtime/commit/6dc320ed049427c93a6d04c51494d1072c376d62) Diff | [Diff](https://github.com/dotnet/runtime/compare/eb02b202c281188e9187d3c292cc223107d35a25...6dc320ed049427c93a6d04c51494d1072c376d62) ### Regressions in System.Globalization.Tests.StringSearch Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IsPrefix_FirstHalf - Duration of single invocation]() | 206.66 ns | 229.92 ns | 1.11 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 157.66 ns | 181.28 ns | 1.15 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 157.44 ns | 181.38 ns | 1.15 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 157.45 ns | 181.39 ns | 1.15 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 206.70 ns | 229.91 ns | 1.11 | 0.01 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*' ``` ### Related Issues #### Regressions * [[Perf] Alpine/x64: 2 Regressions on 10/10/2022 3:25:05 PM]() #### Improvements
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 229.92249249794838 > 216.9886323646198. IsChangePoint: Marked as a change because one of 8/15/2022 5:23:15 PM, 8/17/2022 6:09:21 PM, 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -511.1758886379049 (T) = (0 -229.91438151755705) / Math.Sqrt((0.09403051199595515 / (46)) + (0.00013157600581682603 / (2))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (2) - 2, .025) and -0.11374256555709585 = (206.43404376176778 - 229.91438151755705) / 206.43404376176778 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.2754494076922 > 165.27447788978907. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -587.1745781288312 (T) = (0 -181.2439522304857) / Math.Sqrt((0.03207028389040226 / (49)) + (0.0019841443439575675 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1513574089450762 = (157.41762794278563 - 181.2439522304857) / 157.41762794278563 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.37783263034405 > 165.30027593373453. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -206.95138528770087 (T) = (0 -181.49162825910662) / Math.Sqrt((0.02835066301893864 / (49)) + (0.025898890250939163 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1529053477261196 = (157.42110019444647 - 181.49162825910662) / 157.42110019444647 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.38782301127938 > 165.30941049594205. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -1069.165213210168 (T) = (0 -181.38718382011413) / Math.Sqrt((0.024614597945961592 / (49)) + (8.171306914636648E-07 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1522914146690745 = (157.41433244315766 - 181.38718382011413) / 157.41433244315766 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 229.9117129107842 > 216.87844739024908. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -197.4211601971299 (T) = (0 -230.00988660654446) / Math.Sqrt((0.23150453470920937 / (50)) + (0.01927614907845868 / (2))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (2) - 2, .025) and -0.11423760004232615 = (206.42804245504473 - 230.00988660654446) / 206.42804245504473 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: DrewScoggins
Assignees: -
Labels: `area-System.Memory`, `os-linux`, `tenet-performance`, `tenet-performance-benchmarks`, `arch-x64`, `untriaged`, `os-linux-musl`
Milestone: -
adamsitnik commented 2 years ago

Seems related to https://github.com/dotnet/runtime/pull/73475

I've profiled one of the mentioned test cases:

py .\scripts\benchmarks_ci.py -f net7.0 --filter "System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False))" --bdn-arguments "--profiler ETW"

Vast majority of CPU exclusive time is spent in StartsWithOrdinalIgnoreCaseHelper which is not using SpanHelpers.SequenceCompareTo:

Name                                                                    Exc %   ExcInc %     Inc
System.Private.CoreLib.il!CompareInfo.StartsWithOrdinalIgnoreCaseHelper 83.8 3,238 83.9   3,241
System.Private.CoreLib.il!CompareInfo.IcuStartsWith                       5.7   219 89.6   3,461
microbenchmarks!StringSearch.IsPrefix_FirstHalf                           2.5    95 96.9   3,741
System.Private.CoreLib.il!CompareInfo.IsPrefix                            2.3    87 93.0   3,593
367a4b5d-3b4d-4e52-8b0c-12f54112f6b2!Runnable_0.WorkloadActionUnroll      1.7    65 98.5   3,806
System.Private.CoreLib.il!CultureInfo.get_CompareInfo                     1.3    52  1.3      52
coreclr!JIT_InitPInvokeFrame                                              1.2    45  1.2      45

Since this method was not modified for a long time (months) it's most likely a codegen issue so I am moving it to codegen area.

ghost commented 2 years ago

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [eb02b202c281188e9187d3c292cc223107d35a25](https://github.com/dotnet/runtime/commit/eb02b202c281188e9187d3c292cc223107d35a25) Compare | [6dc320ed049427c93a6d04c51494d1072c376d62](https://github.com/dotnet/runtime/commit/6dc320ed049427c93a6d04c51494d1072c376d62) Diff | [Diff](https://github.com/dotnet/runtime/compare/eb02b202c281188e9187d3c292cc223107d35a25...6dc320ed049427c93a6d04c51494d1072c376d62) ### Regressions in System.Globalization.Tests.StringSearch Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IsPrefix_FirstHalf - Duration of single invocation]() | 206.66 ns | 229.92 ns | 1.11 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 157.66 ns | 181.28 ns | 1.15 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 157.44 ns | 181.38 ns | 1.15 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 157.45 ns | 181.39 ns | 1.15 | 0.01 | False | | | [IsPrefix_FirstHalf - Duration of single invocation]() | 206.70 ns | 229.91 ns | 1.11 | 0.01 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*' ``` ### Related Issues #### Regressions * [[Perf] Alpine/x64: 2 Regressions on 10/10/2022 3:25:05 PM]() #### Improvements
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 229.92249249794838 > 216.9886323646198. IsChangePoint: Marked as a change because one of 8/15/2022 5:23:15 PM, 8/17/2022 6:09:21 PM, 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -511.1758886379049 (T) = (0 -229.91438151755705) / Math.Sqrt((0.09403051199595515 / (46)) + (0.00013157600581682603 / (2))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (2) - 2, .025) and -0.11374256555709585 = (206.43404376176778 - 229.91438151755705) / 206.43404376176778 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.2754494076922 > 165.27447788978907. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -587.1745781288312 (T) = (0 -181.2439522304857) / Math.Sqrt((0.03207028389040226 / (49)) + (0.0019841443439575675 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1513574089450762 = (157.41762794278563 - 181.2439522304857) / 157.41762794278563 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.37783263034405 > 165.30027593373453. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -206.95138528770087 (T) = (0 -181.49162825910662) / Math.Sqrt((0.02835066301893864 / (49)) + (0.025898890250939163 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1529053477261196 = (157.42110019444647 - 181.49162825910662) / 157.42110019444647 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 181.38782301127938 > 165.30941049594205. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -1069.165213210168 (T) = (0 -181.38718382011413) / Math.Sqrt((0.024614597945961592 / (49)) + (8.171306914636648E-07 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1522914146690745 = (157.41433244315766 - 181.38718382011413) / 157.41433244315766 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 229.9117129107842 > 216.87844739024908. IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM. IsRegressionStdDev: Marked as regression because -197.4211601971299 (T) = (0 -230.00988660654446) / Math.Sqrt((0.23150453470920937 / (50)) + (0.01927614907845868 / (2))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (2) - 2, .025) and -0.11423760004232615 = (206.42804245504473 - 230.00988660654446) / 206.42804245504473 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: DrewScoggins
Assignees: -
Labels: `os-linux`, `tenet-performance`, `tenet-performance-benchmarks`, `arch-x64`, `area-CodeGen-coreclr`, `untriaged`, `os-linux-musl`
Milestone: -
JulieLeeMSFT commented 2 years ago

cc @dotnet/jit-contrib @EgorBo.

Since this method was not modified for a long time (months) it's most likely a codegen issue so I am moving it to codegen area.