search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 4/6/2022 2:58:43 AM #4554

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare 22870c2e31ae3c028cece52babd3cae57b560713
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22%5cp%7bLu%7d%22%2c%20Options%3a%20None).html>) 1.95 ms 2.32 ms 1.19 0.04 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22%5cp%7bLu%7d%22%2c%20Options%3a%20NonBacktracking).html>) 2.25 ms 2.62 ms 1.17 0.05 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22Sherlock%7cHolmes%7cWatson%7cIrene%7cAdler%7cJohn%7cBaker%22%2c%20Options%3a%20None).html>) 2.37 ms 2.64 ms 1.12 0.07 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.3180949761904763 > 2.0458638453645834. IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -69.09126502557815 (T) = (0 -2316213.0442670337) / Math.Sqrt((77430135.88498832 / (23)) + (839966103.9571266 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.1877098881191969 = (1950150.5101846738 - 2316213.0442670337) / 1950150.5101846738 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.6243591011904766 > 2.367380766031468. IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 2/18/2022 5:51:51 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -95.24082202415998 (T) = (0 -2609744.036471725) / Math.Sqrt((198229054.07292584 / (23)) + (181395828.47417796 / (33))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (33) - 2, .025) and -0.15889804378927244 = (2251918.5794279124 - 2609744.036471725) / 2251918.5794279124 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.644205665277778 > 2.4892416256250005. IsChangePoint: Marked as a change because one of 3/4/2022 7:13:54 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -75.33269370096944 (T) = (0 -2665699.026560805) / Math.Sqrt((158621225.2102269 / (23)) + (303578297.3573602 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.1266607940049526 = (2366017.3858406995 - 2665699.026560805) / 2366017.3858406995 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare 22870c2e31ae3c028cece52babd3cae57b560713
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern%3a%20%22%5cp%7bSm%7d%22%2c%20Options%3a%20NonBacktracking).html>) 30.24 ms 42.25 ms 1.40 0.04 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern%3a%20%22%5cp%7bSm%7d%22%2c%20Options%3a%20None).html>) 30.20 ms 42.22 ms 1.40 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 42.24819074444444 > 31.75021008083334. IsChangePoint: Marked as a change because one of 3/4/2022 7:13:54 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -345.28566347529625 (T) = (0 -42234592.68980285) / Math.Sqrt((27386450229.901615 / (23)) + (746154687.4875 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.39800516645647593 = (30210612.738188144 - 42234592.68980285) / 30210612.738188144 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 42.222816785714286 > 31.750201288750002. IsChangePoint: Marked as a change because one of 2/26/2022 10:27:55 AM, 3/4/2022 7:13:54 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -1547.1031990324477 (T) = (0 -42239933.26479835) / Math.Sqrt((761649058.7377872 / (23)) + (918922123.5552613 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.3967344504700406 = (30241921.254669003 - 42239933.26479835) / 30241921.254669003 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare 22870c2e31ae3c028cece52babd3cae57b560713
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern%3a%20%22(%3f%3a(%3f%3a25%5b0-5%5d%7c2%5b0-4%5d%5b0-9%5d%7c%5b01%5d%3f%5b0-9%5d%5b0-9%5d)%5c.)%7b3%7d(%3f%3a25%5b0-5%5d%7c2%5b0-4%5d%5b0-9%5d%7c%5b01%5d%3f%5b0-9%5d%5b0-9%5d)%22%2c%20Options%3a%20None).html>) 24.18 ms 27.84 ms 1.15 0.02 False
[Ctor - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern%3a%20%22%5b%5cw%5d%2b%3a%2f%2f%5b%5e%2f%5cs%3f%23%5d%2b%5b%5e%5cs%3f%23%5d%2b(%3f%3a%5c%3f%5b%5e%5cs%23%5d)%3f(%3f%3a%23%5b%5e%5cs%5d)%3f%22%2c%20Options%3a%20NonBacktracking).html>) 285.94 μs 1.95 ms 6.83 0.04 True
[Ctor - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern%3a%20%22%5b%5cw%5c.%2b-%5d%2b%40%5b%5cw%5c.-%5d%2b%5c.%5b%5cw%5c.-%5d%2b%22%2c%20Options%3a%20NonBacktracking).html>) 249.03 μs 1.47 ms 5.91 0.07 True
[Ctor - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern%3a%20%22(%3f%3a(%3f%3a25%5b0-5%5d%7c2%5b0-4%5d%5b0-9%5d%7c%5b01%5d%3f%5b0-9%5d%5b0-9%5d)%5c.)%7b3%7d(%3f%3a25%5b0-5%5d%7c2%5b0-4%5d%5b0-9%5d%7c%5b01%5d%3f%5b0-9%5d%5b0-9%5d)%22%2c%20Options%3a%20NonBacktracking).html>) 66.48 μs 244.18 μs 3.67 0.01 True
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern%3a%20%22(%3f%3a(%3f%3a25%5b0-5%5d%7c2%5b0-4%5d%5b0-9%5d%7c%5b01%5d%3f%5b0-9%5d%5b0-9%5d)%5c.)%7b3%7d(%3f%3a25%5b0-5%5d%7c2%5b0-4%5d%5b0-9%5d%7c%5b01%5d%3f%5b0-9%5d%5b0-9%5d)%22%2c%20Options%3a%20NonBacktracking).html>) 16.86 ms 20.96 ms 1.24 0.04 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.835755933333335 > 25.553284968750003. IsChangePoint: Marked as a change because one of 2/4/2022 6:00:25 PM, 2/21/2022 9:58:09 PM, 2/26/2022 10:27:55 AM, 3/4/2022 7:13:54 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -48.558350558491995 (T) = (0 -27930603.58269769) / Math.Sqrt((59918978874.59263 / (23)) + (76576414633.32838 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.13787336915746864 = (24546319.774913736 - 27930603.58269769) / 24546319.774913736 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "[\w]+://[^/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?", Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.95169642377261 > 296.6648672522152. IsChangePoint: Marked as a change because one of 4/7/2022 6:53:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -361.180655584455 (T) = (0 -1951421.1984782545) / Math.Sqrt((31955207.747185376 / (33)) + (489110493.6088338 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -5.904555680939167 = (282628.06307223806 - 1951421.1984782545) / 282628.06307223806 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.4725886280642435 > 262.3406800099254. IsChangePoint: Marked as a change because one of 4/7/2022 6:53:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -447.3907700394565 (T) = (0 -1476079.7806305643) / Math.Sqrt((32155071.82372137 / (32)) + (156045804.10504302 / (24))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (24) - 2, .025) and -4.8972871989354125 = (250298.09992924012 - 1476079.7806305643) / 250298.09992924012 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 244.17867809152875 > 70.72003371335074. IsChangePoint: Marked as a change because one of 4/7/2022 6:53:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -374.2641489162463 (T) = (0 -242737.65345297105) / Math.Sqrt((592366.6223037967 / (32)) + (4822234.862072869 / (24))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (24) - 2, .025) and -2.6005617510594816 = (67416.60613973484 - 242737.65345297105) / 67416.60613973484 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 20.955751505555558 > 17.61400791423077. IsChangePoint: Marked as a change because one of 2/21/2022 9:58:09 PM, 3/4/2022 7:13:54 PM, 3/15/2022 12:53:39 AM, 3/17/2022 6:54:53 PM, 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM. IsRegressionStdDev: Marked as regression because -166.0432711346411 (T) = (0 -20920673.809335724) / Math.Sqrt((12008787839.406961 / (23)) + (3591530003.0019236 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.24821634497893985 = (16760454.943160275 - 20920673.809335724) / 16760454.943160275 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
AndyAyersMS commented 2 years ago

Some of these are since fixed; per https://github.com/dotnet/runtime/pull/67673 the regression in the .Ctor variants is expected.