search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.19k stars 4.72k forks source link

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock #66768

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 801e81ee8a5d934a50d91d5dca9ad62a4c708bff
Compare 5ce2b9f860f7a84c3059650bb67817d59d8f4953
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%7cHolmes%7cWatson%7cIrene%7cAdler%7cJohn%7cBaker%22%2c%20Options%3a%20Compiled).html>) 3.30 ms 3.50 ms 1.06 0.00 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.5030039285714287 > 3.4672446442307687. IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 2/8/2022 5:20:12 PM, 3/10/2022 4:49:34 PM, 3/17/2022 12:08:34 AM falls between 3/8/2022 5:51:58 AM and 3/17/2022 12:08:34 AM. IsRegressionStdDev: Marked as regression because -149.07005023991758 (T) = (0 -3505062.4336392777) / Math.Sqrt((19718548.140993573 / (20)) + (19401302.82784744 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06171162112192499 = (3301331.8907967033 - 3505062.4336392777) / 3301331.8907967033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 2 years ago

Can be one of https://github.com/dotnet/runtime/pull/66432, https://github.com/dotnet/runtime/pull/66339 or https://github.com/dotnet/runtime/pull/66363 cc @stephentoub

stephentoub commented 2 years ago

Based on the specific pattern, it would almost certainly be the switch to use StartsWith(..., OrdinalIgnoreCase) (https://github.com/dotnet/runtime/pull/66339), since this pattern would have us doing a comparison against 7 different strings case-insensitive. @EgorBo, is your change to lift the inlining/unrolling limit reflected here?

EgorBo commented 2 years ago

@stephentoub if it's StartsWith we need to wait a couple of days to see if https://github.com/dotnet/runtime/pull/66534 helped (https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_arm64_Windows%2010.0.19041%2fSystem.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%7cHolmes%7cWatson%22%2c%20Options%3a%20Compiled).html).

It worth noting that my PR to inline StartsWith handles only up to 16 chars for arm64 where we don't have Vector256. But I have a small patch that extended that up to 64 chars for hot blocks (will file a PR this weekend)

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-text-json See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [801e81ee8a5d934a50d91d5dca9ad62a4c708bff](https://github.com/dotnet/runtime/commit/801e81ee8a5d934a50d91d5dca9ad62a4c708bff) Compare | [5ce2b9f860f7a84c3059650bb67817d59d8f4953](https://github.com/dotnet/runtime/commit/5ce2b9f860f7a84c3059650bb67817d59d8f4953) Diff | [Diff](https://github.com/dotnet/runtime/compare/801e81ee8a5d934a50d91d5dca9ad62a4c708bff...5ce2b9f860f7a84c3059650bb67817d59d8f4953) ### Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Count - Duration of single invocation]() | 3.30 ms | 3.50 ms | 1.06 | 0.00 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.5030039285714287 > 3.4672446442307687. IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 2/8/2022 5:20:12 PM, 3/10/2022 4:49:34 PM, 3/17/2022 12:08:34 AM falls between 3/8/2022 5:51:58 AM and 3/17/2022 12:08:34 AM. IsRegressionStdDev: Marked as regression because -149.07005023991758 (T) = (0 -3505062.4336392777) / Math.Sqrt((19718548.140993573 / (20)) + (19401302.82784744 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06171162112192499 = (3301331.8907967033 - 3505062.4336392777) / 3301331.8907967033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `area-System.Text.Json`, `untriaged`, `refs/heads/main`, `RunKind=micro`, `Windows 10.0.19041`, `Regression`, `CoreClr`, `arm64`
Milestone: -
ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [801e81ee8a5d934a50d91d5dca9ad62a4c708bff](https://github.com/dotnet/runtime/commit/801e81ee8a5d934a50d91d5dca9ad62a4c708bff) Compare | [5ce2b9f860f7a84c3059650bb67817d59d8f4953](https://github.com/dotnet/runtime/commit/5ce2b9f860f7a84c3059650bb67817d59d8f4953) Diff | [Diff](https://github.com/dotnet/runtime/compare/801e81ee8a5d934a50d91d5dca9ad62a4c708bff...5ce2b9f860f7a84c3059650bb67817d59d8f4953) ### Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Count - Duration of single invocation]() | 3.30 ms | 3.50 ms | 1.06 | 0.00 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.5030039285714287 > 3.4672446442307687. IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 2/8/2022 5:20:12 PM, 3/10/2022 4:49:34 PM, 3/17/2022 12:08:34 AM falls between 3/8/2022 5:51:58 AM and 3/17/2022 12:08:34 AM. IsRegressionStdDev: Marked as regression because -149.07005023991758 (T) = (0 -3505062.4336392777) / Math.Sqrt((19718548.140993573 / (20)) + (19401302.82784744 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06171162112192499 = (3301331.8907967033 - 3505062.4336392777) / 3301331.8907967033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `area-System.Text.RegularExpressions`, `untriaged`, `refs/heads/main`, `RunKind=micro`, `Windows 10.0.19041`, `Regression`, `CoreClr`, `arm64`
Milestone: -
am11 commented 2 years ago

arm64 is a redundant label, it can be deleted (I've updated the two issues it was assigned to).

danmoseley commented 2 years ago

I deleted the label. These things pop up when issues are ported to this repo I believe.

stephentoub commented 2 years ago

The regression is gone.

image