search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/x64: 25 Improvements on 5/1/2023 6:56:14 PM #17243

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 0c419b05455970ea4b009c14984785b59ba7b624
Compare da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options%3a%20IgnoreCase%2c%20Compiled).html>) 2.22 secs 1.03 secs 0.46 0.00 False 8360714285.714286 6336000000 0.7578299871849636) Trace Trace
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options%3a%20IgnoreCase).html>) 2.63 secs 1.19 secs 0.45 0.01 False 17793666666.666668 8786333333.333334 0.493789925254304) Trace Trace
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options%3a%20IgnoreCase%2c%20NonBacktracking).html>) 2.62 secs 1.11 secs 0.42 0.02 False Trace Trace

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.0295242 < 2.094407154895834. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 405.6772640223495 (T) = (0 -1026931850) / Math.Sqrt((501668036180779.25 / (299)) + (13440557045000 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5337560006771512 = (2202563146.1026163 - 1026931850) / 2202563146.1026163 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__Compi_baseline_9ebf4757-6be4-4979-b57f-05ca62ad62fb.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__Compi_compare_9ebf4757-6be4-4979-b57f-05ca62ad62fb.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__Compi_baseline_9ebf4757-6be4-4979-b57f-05ca62ad62fb.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__Compi_compare_9ebf4757-6be4-4979-b57f-05ca62ad62fb.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1913559 < 2.4863731329464285. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 45.50413471891439 (T) = (0 -1159110925) / Math.Sqrt((735345849129441.4 / (299)) + (2079476825501250 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5589589662685562 = (2628124905.2799 - 1159110925) / 2628124905.2799 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__baseline_d0f20f9a-d515-4f66-8a02-4d98e1c5cf94.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__compare_d0f20f9a-d515-4f66-8a02-4d98e1c5cf94.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__baseline_d0f20f9a-d515-4f66-8a02-4d98e1c5cf94.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__compare_d0f20f9a-d515-4f66-8a02-4d98e1c5cf94.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1096844857142856 < 2.483436103333333. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 384.8281765418956 (T) = (0 -1111826671.4285712) / Math.Sqrt((3345871140272212 / (299)) + (9177919268979.738 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5789300093356594 = (2640479483.409382 - 1111826671.4285712) / 2640479483.409382 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__NonBa_baseline_ce96d490-7355-49ab-9a3e-06057bffc5dc.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__NonBa_compare_ce96d490-7355-49ab-9a3e-06057bffc5dc.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__NonBa_baseline_ce96d490-7355-49ab-9a3e-06057bffc5dc.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_SliceSlice_Count_Options__IgnoreCase__NonBa_compare_ce96d490-7355-49ab-9a3e-06057bffc5dc.log) ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 0c419b05455970ea4b009c14984785b59ba7b624
Compare da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern%3a%20%22(%3fi)Twain%22%2c%20Options%3a%20None).html>) 8.63 ms 4.03 ms 0.47 0.04 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern%3a%20%22(%3fi)Twain%22%2c%20Options%3a%20NonBacktracking).html>) 8.60 ms 3.87 ms 0.45 0.02 False 47331896.551724136 20548809.523809522 0.4341429568822338)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern%3a%20%22(%3fi)Twain%22%2c%20Options%3a%20Compiled).html>) 7.32 ms 3.90 ms 0.53 0.01 False 22175757.575757574 19689583.333333332 0.88788774255261)

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 4.029435267857143 < 8.13835538690476. IsChangePoint: Marked as a change because one of 3/26/2023 1:36:36 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 43.329972635590664 (T) = (0 -3921212.165178572) / Math.Sqrt((125099360098.54428 / (299)) + (23424479906.753 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.548950965479832 = (8693538.540327461 - 3921212.165178572) / 8693538.540327461 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_baseline_fc127f78-ed11-4404-a7fe-31d38cc90824.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_compare_fc127f78-ed11-4404-a7fe-31d38cc90824.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_baseline_fc127f78-ed11-4404-a7fe-31d38cc90824.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_compare_fc127f78-ed11-4404-a7fe-31d38cc90824.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.8746594354838715 < 8.17683475053214. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 32.67422191755519 (T) = (0 -3722211.1463133637) / Math.Sqrt((15753120006.292988 / (299)) + (46480961742.02939 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5726009536729092 = (8708983.275233457 - 3722211.1463133637) / 8708983.275233457 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_baseline_17f10742-afb5-4085-9e49-1d3a55a4d652.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_compare_17f10742-afb5-4085-9e49-1d3a55a4d652.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_baseline_17f10742-afb5-4085-9e49-1d3a55a4d652.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_compare_17f10742-afb5-4085-9e49-1d3a55a4d652.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.8972188368055556 < 6.947657220588235. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 19.092647344387327 (T) = (0 -3708667.816840278) / Math.Sqrt((12068361907.064415 / (299)) + (71102974259.89313 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.49270333446278924 = (7310648.91371387 - 3708667.816840278) / 7310648.91371387 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_baseline_b1d78df9-5d20-478f-9d5b-6749263d34a3.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_compare_b1d78df9-5d20-478f-9d5b-6749263d34a3.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_baseline_b1d78df9-5d20-478f-9d5b-6749263d34a3.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_Leipzig_Count_Pattern_____i_Twain___Options_compare_b1d78df9-5d20-478f-9d5b-6749263d34a3.log) ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 0c419b05455970ea4b009c14984785b59ba7b624
Compare da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)the%22%2c%20Options%3a%20Compiled).html>) 905.30 μs 603.83 μs 0.67 0.00 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Holmes%22%2c%20Options%3a%20Compiled).html>) 472.29 μs 126.51 μs 0.27 0.01 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%22%2c%20Options%3a%20None).html>) 160.16 μs 84.42 μs 0.53 0.03 False 1149945.067018238 717204.1166380789 0.623685545691118)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%20Holmes%22%2c%20Options%3a%20None).html>) 167.76 μs 97.07 μs 0.58 0.02 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)the%22%2c%20Options%3a%20None).html>) 1.44 ms 970.17 μs 0.67 0.01 False 12474698.795180723 9495201.037613489 0.7611567376105076)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%22%2c%20Options%3a%20Compiled).html>) 124.72 μs 81.29 μs 0.65 0.02 False 725361.4035087719 618754.6369899311 0.8530294471098756)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Holmes%22%2c%20Options%3a%20None).html>) 559.23 μs 168.93 μs 0.30 0.00 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)the%22%2c%20Options%3a%20NonBacktracking).html>) 1.63 ms 1.18 ms 0.72 0.01 False 13480610.021786492 10509268.292682927 0.779584030373887)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Holmes%22%2c%20Options%3a%20NonBacktracking).html>) 554.71 μs 171.16 μs 0.31 0.01 False 4842910.170749814 1280464.9711343423 0.2643999012965569)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%22%2c%20Options%3a%20NonBacktracking).html>) 161.57 μs 93.84 μs 0.58 0.01 False 1125399.9561691869 710058.7625302455 0.6309390351739972)
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%20Holmes%22%2c%20Options%3a%20NonBacktracking).html>) 161.99 μs 95.40 μs 0.59 0.03 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%20Holmes%22%2c%20Options%3a%20Compiled).html>) 122.91 μs 74.81 μs 0.61 0.03 False 730261.2994350282 617054.1465378421 0.8449771979087901)

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 603.8255362426036 < 867.5803859953704. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 164.30312600529408 (T) = (0 -605649.0261341224) / Math.Sqrt((12439218.218503278 / (299)) + (6650230.768942348 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.3323404720685283 = (907122.5689095325 - 605649.0261341224) / 907122.5689095325 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___baseline_4999285d-a569-4f5b-b955-421c663c0e35.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___compare_4999285d-a569-4f5b-b955-421c663c0e35.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___baseline_4999285d-a569-4f5b-b955-421c663c0e35.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___compare_4999285d-a569-4f5b-b955-421c663c0e35.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 126.50510924796751 < 449.5628845492742. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 99.63093064968355 (T) = (0 -123001.35657807806) / Math.Sqrt((18274719.023708306 / (299)) + (24552565.543514907 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.7399281255539418 = (472951.39791669376 - 123001.35657807806) / 472951.39791669376 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_baseline_f9ac9012-5660-49df-a6d0-1c75fb7fd9f3.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_compare_f9ac9012-5660-49df-a6d0-1c75fb7fd9f3.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_baseline_f9ac9012-5660-49df-a6d0-1c75fb7fd9f3.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_compare_f9ac9012-5660-49df-a6d0-1c75fb7fd9f3.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 84.4211189516129 < 154.45343088860542. IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 18.850700841846812 (T) = (0 -88481.44086221518) / Math.Sqrt((6640206.706227603 / (299)) + (32972428.03543391 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.46398562972012336 = (165072.8894003628 - 88481.44086221518) / 165072.8894003628 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_36558bd7-4589-4564-a149-a7b505307b48.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_36558bd7-4589-4564-a149-a7b505307b48.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_36558bd7-4589-4564-a149-a7b505307b48.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_36558bd7-4589-4564-a149-a7b505307b48.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 97.06518487394958 < 158.0798341621864. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 18.813246580250425 (T) = (0 -93196.97809038387) / Math.Sqrt((5827557.995732645 / (299)) + (29926047.44084751 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.4386329083399192 = (166017.8864685152 - 93196.97809038387) / 166017.8864685152 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_58c025be-d8cb-413d-b692-34eed311cef4.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_58c025be-d8cb-413d-b692-34eed311cef4.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_58c025be-d8cb-413d-b692-34eed311cef4.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_58c025be-d8cb-413d-b692-34eed311cef4.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 970.1692647058821 < 1.3689006287878787. IsChangePoint: Marked as a change because one of 2/21/2023 5:46:41 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 239.35198479553316 (T) = (0 -971794.8146446077) / Math.Sqrt((397700472.95758665 / (299)) + (5284825.206581594 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.3292653064700413 = (1448851.2731169795 - 971794.8146446077) / 1448851.2731169795 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___baseline_d6c699f9-751c-4166-90fc-563d7f7b37eb.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___compare_d6c699f9-751c-4166-90fc-563d7f7b37eb.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___baseline_d6c699f9-751c-4166-90fc-563d7f7b37eb.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___compare_d6c699f9-751c-4166-90fc-563d7f7b37eb.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 81.28570312499998 < 119.37981758297012. IsChangePoint: Marked as a change because one of 3/17/2023 9:25:32 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 107.47172074984647 (T) = (0 -81684.7298693783) / Math.Sqrt((6368930.429551073 / (299)) + (318444.68545831577 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.3585673784580143 = (127347.32710196519 - 81684.7298693783) / 127347.32710196519 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_ffb53718-d59c-4add-bfad-e8c026b1fb19.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_ffb53718-d59c-4add-bfad-e8c026b1fb19.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_ffb53718-d59c-4add-bfad-e8c026b1fb19.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_ffb53718-d59c-4add-bfad-e8c026b1fb19.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: None) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 168.93247282608695 < 531.6867780257936. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 111.47856004333808 (T) = (0 -165397.96404361184) / Math.Sqrt((26134731.58358133 / (299)) + (24985504.666787367 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.7050656804157615 = (560795.923230532 - 165397.96404361184) / 560795.923230532 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_baseline_a83c19f1-caa7-4d4f-bef2-3e512b575833.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_compare_a83c19f1-caa7-4d4f-bef2-3e512b575833.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_baseline_a83c19f1-caa7-4d4f-bef2-3e512b575833.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_compare_a83c19f1-caa7-4d4f-bef2-3e512b575833.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1810594944707742 < 1.5563478178807948. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 198.4230062437523 (T) = (0 -1182908.1311111269) / Math.Sqrt((616019221.6359057 / (299)) + (6834914.856107898 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.28191438833840615 = (1647307.9419791885 - 1182908.1311111269) / 1647307.9419791885 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___baseline_0f31e89f-36ba-423b-9d05-66e8eb3d6971.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___compare_0f31e89f-36ba-423b-9d05-66e8eb3d6971.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___baseline_0f31e89f-36ba-423b-9d05-66e8eb3d6971.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_the___compare_0f31e89f-36ba-423b-9d05-66e8eb3d6971.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 171.15945861204014 < 526.2841695817866. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 108.97320879384878 (T) = (0 -167612.31394143676) / Math.Sqrt((28532486.243310507 / (299)) + (25164470.62838026 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.6983323382106121 = (555619.0973444707 - 167612.31394143676) / 555619.0973444707 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_baseline_0e6b7438-a9e3-4d3b-bb00-c1bfe09ff928.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_compare_0e6b7438-a9e3-4d3b-bb00-c1bfe09ff928.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_baseline_0e6b7438-a9e3-4d3b-bb00-c1bfe09ff928.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Holme_compare_0e6b7438-a9e3-4d3b-bb00-c1bfe09ff928.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 93.83685254491019 < 153.26225309602845. IsChangePoint: Marked as a change because one of 2/24/2023 4:59:28 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 43.71886320755424 (T) = (0 -92200.10130965748) / Math.Sqrt((6183335.018718086 / (299)) + (5357909.212202556 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.4379162095671564 = (164032.6635974629 - 92200.10130965748) / 164032.6635974629 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_592a5157-a639-410f-b0aa-e9b3d41e0a9a.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_592a5157-a639-410f-b0aa-e9b3d41e0a9a.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_592a5157-a639-410f-b0aa-e9b3d41e0a9a.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_592a5157-a639-410f-b0aa-e9b3d41e0a9a.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: NonBacktracking) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 95.40336775015933 < 155.32739015981596. IsChangePoint: Marked as a change because one of 2/17/2023 9:53:52 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 38.864904013467935 (T) = (0 -93626.98525677223) / Math.Sqrt((9254617.732881743 / (299)) + (6311069.525624342 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.42561431141614375 = (163003.68744842664 - 93626.98525677223) / 163003.68744842664 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_e64eddc1-54b4-4662-b1cf-ecc2769a96a1.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_e64eddc1-54b4-4662-b1cf-ecc2769a96a1.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_e64eddc1-54b4-4662-b1cf-ecc2769a96a1.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_e64eddc1-54b4-4662-b1cf-ecc2769a96a1.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 74.80931364251862 < 121.04706922205007. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 13.140045592343006 (T) = (0 -78508.27962827685) / Math.Sqrt((5808452.345192643 / (299)) + (27364698.72759271 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.38254098562491423 = (127147.3535903157 - 78508.27962827685) / 127147.3535903157 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_5f6549ab-8889-461f-955f-d43bce05de5d.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_5f6549ab-8889-461f-955f-d43bce05de5d.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_baseline_5f6549ab-8889-461f-955f-d43bce05de5d.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Industry_RustLang_Sherlock_Count_Pattern_____i_Sherl_compare_5f6549ab-8889-461f-955f-d43bce05de5d.log) ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 0c419b05455970ea4b009c14984785b59ba7b624
Compare da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[MatchesWord - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options%3a%20IgnoreCase%2c%20Compiled).html>) 2.16 μs 1.19 μs 0.55 0.01 False 14623.96042443361 10745.147479841358 0.7347631672941648) Trace Trace
[ReplaceWords - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options%3a%20Compiled).html>) 3.46 μs 3.11 μs 0.90 0.00 True Trace Trace
[CtorInvoke - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options%3a%20IgnoreCase%2c%20Compiled).html>) 320.04 μs 298.82 μs 0.93 0.00 True Trace Trace
[CtorInvoke - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options%3a%20Compiled).html>) 303.45 μs 278.38 μs 0.92 0.00 True Trace Trace
[ReplaceWords - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options%3a%20IgnoreCase%2c%20Compiled).html>) 3.59 μs 3.25 μs 0.90 0.02 False Trace Trace

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.1885091665399914 < 2.0592669454219905. IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 188.95302163107965 (T) = (0 -1183.496426952629) / Math.Sqrt((810.0671866037898 / (299)) + (50.25511634142212 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.4572172933355851 = (2180.423975232407 - 1183.496426952629) / 2180.423975232407 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_MatchesWord_Options__IgnoreCase__Compiled__baseline_6fc0798a-416c-46f0-a685-b533d11c58f7.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_MatchesWord_Options__IgnoreCase__Compiled__compare_6fc0798a-416c-46f0-a685-b533d11c58f7.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_MatchesWord_Options__IgnoreCase__Compiled__baseline_6fc0798a-416c-46f0-a685-b533d11c58f7.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_MatchesWord_Options__IgnoreCase__Compiled__compare_6fc0798a-416c-46f0-a685-b533d11c58f7.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.1084394916627183 < 3.300970939639454. IsChangePoint: Marked as a change because one of 4/30/2023 6:17:41 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 49.89616814227903 (T) = (0 -3115.5684656227463) / Math.Sqrt((852.2325429893483 / (299)) + (346.57398106565506 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.11086150897605744 = (3504.0305836212538 - 3115.5684656227463) / 3504.0305836212538 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__Compiled__baseline_1c5d3e1a-1874-418e-a659-0dcafb0e0225.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__Compiled__compare_1c5d3e1a-1874-418e-a659-0dcafb0e0225.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__Compiled__baseline_1c5d3e1a-1874-418e-a659-0dcafb0e0225.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__Compiled__compare_1c5d3e1a-1874-418e-a659-0dcafb0e0225.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 298.8157881181318 < 303.65610146029564. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 28.540387840138976 (T) = (0 -292705.5238339882) / Math.Sqrt((12706652.182637556 / (299)) + (20316001.70745228 / (27))) is greater than 1.9673127716992511 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (27) - 2, .975) and 0.07998210992304278 = (318151.99138084595 - 292705.5238339882) / 318151.99138084595 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__IgnoreCase__Compiled__baseline_1c39c224-6e28-49ad-9279-64fd09acc4c1.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__IgnoreCase__Compiled__compare_1c39c224-6e28-49ad-9279-64fd09acc4c1.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__IgnoreCase__Compiled__baseline_1c39c224-6e28-49ad-9279-64fd09acc4c1.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__IgnoreCase__Compiled__compare_1c39c224-6e28-49ad-9279-64fd09acc4c1.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 278.3799561403508 < 288.9261732343178. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 41.95182016001667 (T) = (0 -274032.0728104471) / Math.Sqrt((10164434.227081925 / (299)) + (9802839.741628235 / (26))) is greater than 1.967335607330539 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (26) - 2, .975) and 0.08937652445573172 = (300927.96877069474 - 274032.0728104471) / 300927.96877069474 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__Compiled__baseline_de45af03-83e0-4bd2-851f-59ca0affbee3.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__Compiled__compare_de45af03-83e0-4bd2-851f-59ca0affbee3.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__Compiled__baseline_de45af03-83e0-4bd2-851f-59ca0affbee3.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_CtorInvoke_Options__Compiled__compare_de45af03-83e0-4bd2-851f-59ca0affbee3.log) #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 3.2475194671415606 < 3.3920194978782767. IsChangePoint: Marked as a change because one of 4/10/2023 8:15:35 PM, 4/12/2023 9:47:42 AM, 4/30/2023 6:17:41 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 31.63969074640035 (T) = (0 -3244.9673689033684) / Math.Sqrt((3925.1948665505283 / (299)) + (657.6510345874302 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.09749005458090693 = (3595.492088894957 - 3244.9673689033684) / 3595.492088894957 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__IgnoreCase__Compiled__baseline_8296b430-a212-4e4d-bf81-bf7f4682657c.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__IgnoreCase__Compiled__compare_8296b430-a212-4e4d-bf81-bf7f4682657c.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__IgnoreCase__Compiled__baseline_8296b430-a212-4e4d-bf81-bf7f4682657c.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Text_RegularExpressions_Tests_Perf_Regex_Common_ReplaceWords_Options__IgnoreCase__Compiled__compare_8296b430-a212-4e4d-bf81-bf7f4682657c.log) ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 0c419b05455970ea4b009c14984785b59ba7b624
Compare da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Ctor_ByteArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString%3a%201234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123.html>) 91.93 ns 83.53 ns 0.91 0.00 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 83.52844315269986 < 87.35984552041162. IsChangePoint: Marked as a change because one of 3/25/2023 10:49:06 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 10.47624125674963 (T) = (0 -82.7600225433461) / Math.Sqrt((2.698390863077113 / (299)) + (1.1809404657592328 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.08926284238695738 = (90.87146807565469 - 82.7600225433461) / 90.87146807565469 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Numerics_Tests_Perf_BigInteger_Ctor_ByteArray_numberString__123456789012345678901234567890123_baseline_82cd4ba2-db48-49e6-8aee-95b3c6f001ab.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Numerics_Tests_Perf_BigInteger_Ctor_ByteArray_numberString__123456789012345678901234567890123_compare_82cd4ba2-db48-49e6-8aee-95b3c6f001ab.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Numerics_Tests_Perf_BigInteger_Ctor_ByteArray_numberString__123456789012345678901234567890123_baseline_82cd4ba2-db48-49e6-8aee-95b3c6f001ab.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Numerics_Tests_Perf_BigInteger_Ctor_ByteArray_numberString__123456789012345678901234567890123_compare_82cd4ba2-db48-49e6-8aee-95b3c6f001ab.log) ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 0c419b05455970ea4b009c14984785b59ba7b624
Compare da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Improvements in System.Collections.Sort<IntStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[List - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Collections.Sort(IntStruct).List(Size%3a%20512).html>) 7.63 μs 4.62 μs 0.61 0.30 False Trace Trace

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;IntStruct&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Sort<IntStruct>.List(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 4.619933333333333 < 7.113910921428571. IsChangePoint: Marked as a change because one of 2/24/2023 12:59:32 PM, 3/6/2023 6:03:21 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM. IsImprovementStdDev: Marked as improvement because 62.680420749717165 (T) = (0 -4659.403333333334) / Math.Sqrt((97408.49106954872 / (299)) + (3115.7618000000402 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.36862803269384997 = (7379.807109925116 - 4659.403333333334) / 7379.807109925116 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms [Baseline](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Collections_Sort_IntStruct__List_Size__512__baseline_7acededb-f7b8-4aec-82f6-2f4a59a94ddf.log) [Compare](https://pvscmdupload.z22.web.core.windows.net/autofilereport/jitdasms/05_02_2023/System_Collections_Sort_IntStruct__List_Size__512__compare_7acededb-f7b8-4aec-82f6-2f4a59a94ddf.log) [Diff](https://perfsupport.azurewebsites.net/diff?old=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Collections_Sort_IntStruct__List_Size__512__baseline_7acededb-f7b8-4aec-82f6-2f4a59a94ddf.log&new=https%3A%2F%2Fpvscmdupload.blob.core.windows.net%2Fautofilereport%2Fjitdasms%2F05_02_2023%2FSystem_Collections_Sort_IntStruct__List_Size__512__compare_7acededb-f7b8-4aec-82f6-2f4a59a94ddf.log) ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
AndyAyersMS commented 1 year ago

Either https://github.com/dotnet/runtime/pull/85546 or https://github.com/dotnet/runtime/pull/85398 or https://github.com/dotnet/runtime/pull/85438