search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/x64: 32 Regressions on 12/14/2022 8:02:51 PM #10973

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Memory.Constructors<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ArrayAsMemory - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Constructors(String).ArrayAsMemory.html>) 8.66 ns 10.30 ns 1.19 0.12 False
[ArrayAsMemoryStartLength - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Constructors(String).ArrayAsMemoryStartLength.html>) 10.40 ns 12.07 ns 1.16 0.11 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Constructors&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Constructors<String>.ArrayAsMemory ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.301544918134786 > 9.091656223426604. IsChangePoint: Marked as a change because one of 10/6/2022 2:13:23 PM, 10/7/2022 12:48:42 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -20.771246428110643 (T) = (0 -10.399461758219939) / Math.Sqrt((0.020382174586075018 / (33)) + (0.14041086503439926 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.19137665290024344 = (8.72894540354125 - 10.399461758219939) / 8.72894540354125 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Constructors<String>.ArrayAsMemoryStartLength ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.06629996999254 > 10.92039279485286. IsChangePoint: Marked as a change because one of 11/18/2022 3:21:53 PM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -21.84274089465631 (T) = (0 -11.929565089631211) / Math.Sqrt((0.009308799819848992 / (33)) + (0.1053978264845209 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.14308332127197623 = (10.436304045059883 - 11.929565089631211) / 10.436304045059883 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Backtracking - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options%3a%20IgnoreCase%2c%20Compiled).html>) 116.71 ns 134.48 ns 1.15 0.12 False
[Uri_IsMatch - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options%3a%20IgnoreCase%2c%20Compiled).html>) 87.44 ns 94.71 ns 1.08 0.07 False
[Uri_IsMatch - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options%3a%20Compiled).html>) 85.86 ns 101.60 ns 1.18 0.06 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 134.4754558671903 > 119.50558138176397. IsChangePoint: Marked as a change because one of 10/11/2022 2:16:14 AM, 10/26/2022 2:14:52 PM, 12/6/2022 2:12:43 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -50.171616562646584 (T) = (0 -132.08701880253648) / Math.Sqrt((1.6304113982026887 / (33)) + (1.6295425802294703 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.1495483534487559 = (114.90340393795762 - 132.08701880253648) / 114.90340393795762 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 94.70574666584466 > 92.64085840310346. IsChangePoint: Marked as a change because one of 10/14/2022 3:33:41 AM, 10/18/2022 1:35:00 AM, 11/14/2022 1:59:26 PM, 11/16/2022 6:51:59 PM, 12/7/2022 6:04:36 PM, 12/14/2022 12:26:48 AM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -9.99993845300169 (T) = (0 -92.71195709377086) / Math.Sqrt((2.997955635176172 / (29)) + (3.903114628057193 / (28))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (28) - 2, .025) and -0.056128042155565834 = (87.78476983202246 - 92.71195709377086) / 87.78476983202246 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 101.59557220936861 > 90.05438982315187. IsChangePoint: Marked as a change because one of 11/8/2022 2:29:58 AM, 11/11/2022 5:53:56 PM, 12/5/2022 9:44:53 PM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -19.392417028966364 (T) = (0 -97.88503201553097) / Math.Sqrt((3.4334895128441247 / (33)) + (6.344459847661725 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.136682617664026 = (86.11465548465284 - 97.88503201553097) / 86.11465548465284 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SequenceCompareTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).SequenceCompareTo(Size%3a%20512).html>) 13.18 ns 14.71 ns 1.12 0.01 False
[SequenceCompareToDifferent - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).SequenceCompareToDifferent(Size%3a%2033).html>) 5.16 ns 6.24 ns 1.21 0.03 False
[SequenceCompareTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).SequenceCompareTo(Size%3a%2033).html>) 5.11 ns 6.24 ns 1.22 0.01 False
[SequenceCompareToDifferent - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).SequenceCompareToDifferent(Size%3a%20512).html>) 5.15 ns 6.19 ns 1.20 0.02 False
[SequenceCompareTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).SequenceCompareTo(Size%3a%204).html>) 6.62 ns 7.77 ns 1.17 0.10 False
[SequenceCompareToDifferent - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).SequenceCompareToDifferent(Size%3a%204).html>) 4.98 ns 6.01 ns 1.21 0.03 False
[IndexOfAnyTwoValues - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Byte).IndexOfAnyTwoValues(Size%3a%20512).html>) 7.65 ns 8.93 ns 1.17 0.02 False

graph graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Byte>.SequenceCompareTo(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 14.711578720633897 > 13.861916801051576. IsChangePoint: Marked as a change because one of 11/10/2022 9:13:35 AM, 11/11/2022 5:53:56 PM, 11/22/2022 2:20:42 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -33.77723597787299 (T) = (0 -15.011218015943028) / Math.Sqrt((0.00957341554156296 / (33)) + (0.05879383016933273 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.13350402249868473 = (13.243197834315975 - 15.011218015943028) / 13.243197834315975 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 33) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.238958635916576 > 5.500810245973619. IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -46.71503518765774 (T) = (0 -6.3187305497495) / Math.Sqrt((0.00631421130092891 / (33)) + (0.00820878263949395 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.20589830791435595 = (5.239853566656022 - 6.3187305497495) / 5.239853566656022 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.SequenceCompareTo(Size: 33) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.243031860162625 > 5.536209612426748. IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -51.811795818729934 (T) = (0 -6.3757725452131675) / Math.Sqrt((0.005575180846440842 / (33)) + (0.0074525437259974035 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.2164573380079248 = (5.241262760315259 - 6.3757725452131675) / 5.241262760315259 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.189513757852308 > 5.498759290775134. IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/11/2022 5:53:56 PM, 11/16/2022 6:51:59 PM, 11/18/2022 7:31:41 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -55.83519241154571 (T) = (0 -6.3444156001291825) / Math.Sqrt((0.004200630305148702 / (33)) + (0.006475619731576318 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.21267465176874475 = (5.2317541154798155 - 6.3444156001291825) / 5.2317541154798155 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.SequenceCompareTo(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.770928306804273 > 7.237085478986349. IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -19.168323100293257 (T) = (0 -8.150537633659955) / Math.Sqrt((0.014820299351302203 / (33)) + (0.10459397653366277 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.19482648894783336 = (6.821524053117817 - 8.150537633659955) / 6.821524053117817 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.005121548008978 > 5.487537971702298. IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -21.44879027820133 (T) = (0 -6.3958923420714795) / Math.Sqrt((0.014512890457500737 / (33)) + (0.06583725969778421 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.23334918054357354 = (5.185792023028402 - 6.3958923420714795) / 5.185792023028402 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.925377941696487 > 8.021167500373862. IsChangePoint: Marked as a change because one of 10/31/2022 10:23:25 PM, 11/11/2022 5:53:56 PM, 12/12/2022 5:25:39 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -16.649956532111734 (T) = (0 -8.711118146472637) / Math.Sqrt((0.0004255359214510854 / (20)) + (0.1555647072195135 / (37))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (37) - 2, .025) and -0.14187599873198567 = (7.628777692276602 - 8.711118146472637) / 7.628777692276602 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22the%22%2c%20Options%3a%20None).html>) 646.03 μs 722.06 μs 1.12 0.14 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22.*%22%2c%20Options%3a%20Compiled).html>) 1.00 ms 1.11 ms 1.11 0.18 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22the%22%2c%20Options%3a%20Compiled).html>) 409.14 μs 453.02 μs 1.11 0.07 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22the%5c%5cs%2b%5c%5cw%2b%22%2c%20Options%3a%20Compiled).html>) 570.44 μs 609.35 μs 1.07 0.05 False
[Count - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22%5b%5e%5c%5cn%5d*%22%2c%20Options%3a%20Compiled).html>) 991.80 μs 1.08 ms 1.09 0.11 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 722.0648295454545 > 678.45462125. IsChangePoint: Marked as a change because one of 11/10/2022 5:17:05 PM, 11/22/2022 11:24:47 AM, 12/1/2022 7:29:59 PM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -45.326192212596496 (T) = (0 -714882.7736140484) / Math.Sqrt((13983409.191710504 / (33)) + (40966600.15482388 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.10198837417871436 = (648720.7944882662 - 714882.7736140484) / 648720.7944882662 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1121430952380953 > 1.0574153613281252. IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/10/2022 9:13:35 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -9.183432058444403 (T) = (0 -1120636.846329524) / Math.Sqrt((2203410184.3049884 / (33)) + (1418746864.444214 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.10125438181569123 = (1017600.3517751057 - 1120636.846329524) / 1017600.3517751057 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 453.02236904761907 > 433.56323108552635. IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/19/2022 7:27:15 PM, 10/28/2022 6:19:52 PM, 11/16/2022 6:51:59 PM, 11/18/2022 8:48:12 PM, 12/12/2022 5:25:39 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -13.909840753928085 (T) = (0 -453452.4596847358) / Math.Sqrt((933079.8982790405 / (20)) + (317412291.7993082 / (37))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (37) - 2, .025) and -0.09901042850937564 = (412600.68869389023 - 453452.4596847358) / 412600.68869389023 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\\s+\\w+", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 609.354532967033 > 599.5162566964287. IsChangePoint: Marked as a change because one of 11/11/2022 5:53:56 PM, 11/18/2022 7:31:41 AM, 12/14/2022 12:26:48 AM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -15.736035804535208 (T) = (0 -616088.862026904) / Math.Sqrt((12842397.152561773 / (29)) + (205044322.21990502 / (28))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (28) - 2, .025) and -0.07663269875003502 = (572236.8108846963 - 616088.862026904) / 572236.8108846963 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[^\\n]*", Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.082281953125 > 1044.8415468750002. IsChangePoint: Marked as a change because one of 10/26/2022 7:11:30 AM, 10/26/2022 4:56:52 PM, 12/6/2022 2:12:43 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -18.74533050828114 (T) = (0 -1111681.9019753223) / Math.Sqrt((411995226.2816987 / (33)) + (482512462.84561557 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.10651424415526717 = (1004670.2135533738 - 1111681.9019753223) / 1004670.2135533738 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableSortedDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsKeyFalse(String%2c%20String).ImmutableSortedDictionary(Size%3a%20512).html>) 294.39 μs 320.00 μs 1.09 0.08 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyFalse<String, String>.ImmutableSortedDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 319.999175170068 > 308.15471296296295. IsChangePoint: Marked as a change because one of 11/1/2022 6:58:35 AM, 11/16/2022 11:33:39 PM, 11/18/2022 3:21:53 PM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -22.11161603202125 (T) = (0 -321009.8727602648) / Math.Sqrt((24544966.521344837 / (33)) + (9947698.01260222 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08006798407158244 = (297212.65466099547 - 321009.8727602648) / 297212.65466099547 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IsMatch - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id%3a%2011%2c%20Options%3a%20NonBacktracking).html>) 113.33 ns 125.75 ns 1.11 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 125.75030834789825 > 119.75716389074137. IsChangePoint: Marked as a change because one of 10/24/2022 10:11:47 PM, 10/26/2022 7:11:30 AM, 11/24/2022 9:42:44 PM, 11/29/2022 4:49:06 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -9.580066874130205 (T) = (0 -122.24195725403455) / Math.Sqrt((8.493630338117898 / (33)) + (5.743845565713753 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.05846216254789852 = (115.49015314801368 - 122.24195725403455) / 115.49015314801368 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableSortedSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CtorFromCollection(String).ImmutableSortedSet(Size%3a%20512).html>) 321.71 μs 358.40 μs 1.11 0.13 False
[SortedSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CtorFromCollection(String).SortedSet(Size%3a%20512).html>) 317.07 μs 344.26 μs 1.09 0.15 False
[SortedDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CtorFromCollection(String).SortedDictionary(Size%3a%20512).html>) 300.38 μs 328.70 μs 1.09 0.07 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorFromCollection<String>.ImmutableSortedSet(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 358.39982007575765 > 338.33379832987885. IsChangePoint: Marked as a change because one of 11/1/2022 6:58:35 AM, 11/10/2022 12:29:27 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -7.8656756746344465 (T) = (0 -369559.0544676352) / Math.Sqrt((75132449.86452809 / (33)) + (540850424.2075889 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.11859228209908698 = (330378.6914872518 - 369559.0544676352) / 330378.6914872518 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CtorFromCollection<String>.SortedSet(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 344.2571195652174 > 333.62903090659347. IsChangePoint: Marked as a change because one of 11/20/2022 2:44:51 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -6.404751962621824 (T) = (0 -348143.9555460168) / Math.Sqrt((126099299.49332544 / (33)) + (169087927.66916135 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.06455926323028145 = (327031.0705762067 - 348143.9555460168) / 327031.0705762067 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CtorFromCollection<String>.SortedDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 328.6972916666667 > 315.90151437684915. IsChangePoint: Marked as a change because one of 11/23/2022 8:02:34 PM, 11/29/2022 4:49:06 AM, 12/14/2022 12:26:48 AM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -12.183445146569834 (T) = (0 -327465.71781399735) / Math.Sqrt((62965314.67008396 / (29)) + (25415005.132028665 / (28))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (28) - 2, .025) and -0.0698428246121518 = (306087.68903293146 - 327465.71781399735) / 306087.68903293146 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Ctor_ByteArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString%3a%20-2147483648).html>) 10.97 ns 12.73 ns 1.16 0.04 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString: -2147483648) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.73123175350256 > 11.308895747944185. IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/22/2022 2:20:42 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -60.199194596668995 (T) = (0 -12.803585483575887) / Math.Sqrt((0.03069752764066611 / (33)) + (0.002281677115554566 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.17723394517402058 = (10.875990737493762 - 12.803585483575887) / 10.875990737493762 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableSortedSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsFalse(String).ImmutableSortedSet(Size%3a%20512).html>) 288.00 μs 324.54 μs 1.13 0.05 False
[ImmutableArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsFalse(String).ImmutableArray(Size%3a%20512).html>) 827.34 μs 895.52 μs 1.08 0.01 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsFalse<String>.ImmutableSortedSet(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 324.5404846938775 > 302.30163224957914. IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/10/2022 9:13:35 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -12.274364322932328 (T) = (0 -324089.76036970614) / Math.Sqrt((78740587.93084493 / (33)) + (78409158.93270671 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.09896006476080355 = (294905.8576029754 - 324089.76036970614) / 294905.8576029754 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.ContainsFalse<String>.ImmutableArray(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 895.5220238095237 > 864.8765856085529. IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -93.44758501253024 (T) = (0 -894181.5439814813) / Math.Sqrt((21519887.445277255 / (52)) + (920730.0178849782 / (5))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (5) - 2, .025) and -0.08791969608284549 = (821918.7015374976 - 894181.5439814813) / 821918.7015374976 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CreateAddAndClear(String).SortedDictionary(Size%3a%20512).html>) 295.36 μs 318.61 μs 1.08 0.12 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CreateAddAndClear<String>.SortedDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 318.60916405023545 > 310.0669943985849. IsChangePoint: Marked as a change because one of 11/23/2022 1:02:40 AM, 12/1/2022 7:29:59 PM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -17.464710227626277 (T) = (0 -323003.5103558971) / Math.Sqrt((37027616.72338937 / (33)) + (18860108.620154906 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08071246481424926 = (298880.1562600782 - 323003.5103558971) / 298880.1562600782 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[DeserializeFromUtf8Bytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Json.Serialization.Tests.ReadJson(HashSet(String)).DeserializeFromUtf8Bytes(Mode%3a%20SourceGen).html>) 10.42 μs 11.50 μs 1.10 0.28 False

graph_1.png>) Test Report.html>)

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromUtf8Bytes(Mode: SourceGen) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.500891999887042 > 10.969144693886516. IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -9.862522131716464 (T) = (0 -11663.80816808394) / Math.Sqrt((227615.86031425267 / (33)) + (55523.13281638237 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08831923428666752 = (10717.267324351678 - 11663.80816808394) / 10717.267324351678 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Parse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Decimal.Parse(value%3a%20%22123456.789%22).html>) 79.41 ns 88.85 ns 1.12 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Decimal*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Decimal.Parse(value: "123456.789") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 88.84665554627438 > 81.59114515438536. IsChangePoint: Marked as a change because one of 11/10/2022 9:13:35 AM, 11/11/2022 5:53:56 PM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -10.644086152877717 (T) = (0 -85.27210969007972) / Math.Sqrt((1.822810576559775 / (33)) + (7.195935795163738 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.0803570626539766 = (78.92956193631252 - 85.27210969007972) / 78.92956193631252 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Collections.IterateForEach<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableHashSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.IterateForEach(Int32).ImmutableHashSet(Size%3a%20512).html>) 18.07 μs 20.36 μs 1.13 0.02 False
[ImmutableStack - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.IterateForEach(Int32).ImmutableStack(Size%3a%20512).html>) 1.72 μs 1.87 μs 1.08 0.00 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IterateForEach<Int32>.ImmutableHashSet(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 20.36202835273825 > 18.9993896762961. IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -23.5479736823076 (T) = (0 -20610.686295640407) / Math.Sqrt((14836.353712574088 / (33)) + (250395.45834059207 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.13531516316764722 = (18154.15398675241 - 20610.686295640407) / 18154.15398675241 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.IterateForEach<Int32>.ImmutableStack(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.8651340441205753 > 1.805965732328783. IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -266.91170487633195 (T) = (0 -1863.5895076656561) / Math.Sqrt((0.848436771906827 / (53)) + (1.0954824378070889 / (4))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (53) + (4) - 2, .025) and -0.08355586406846709 = (1719.8831822739328 - 1863.5895076656561) / 1719.8831822739328 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Concat_str_str_str - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_String.Concat_str_str_str(size%3a%20100).html>) 41.86 ns 47.86 ns 1.14 0.07 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_String.Concat_str_str_str(size: 100) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 47.86177376706385 > 40.62773136466727. IsChangePoint: Marked as a change because one of 11/22/2022 2:20:42 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -17.22462234954208 (T) = (0 -46.506595930133756) / Math.Sqrt((3.0834448723476973 / (33)) + (0.5467821336460489 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.1445091873476096 = (40.63453264006766 - 46.506595930133756) / 40.63453264006766 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8e15d9f523209caf08db57897777b7beb0516c99
Compare a78811401f3b2fa8be2d8e40b9bb1e706a9e8501
Diff Diff

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableSortedDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.TryGetValueFalse(String%2c%20String).ImmutableSortedDictionary(Size%3a%20512).html>) 291.05 μs 342.41 μs 1.18 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;String, String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.TryGetValueFalse<String, String>.ImmutableSortedDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 342.4107065217391 > 306.51546932870366. IsChangePoint: Marked as a change because one of 11/2/2022 4:14:20 AM, 11/17/2022 7:01:39 PM, 12/6/2022 2:12:43 AM, 12/14/2022 5:20:21 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM. IsRegressionStdDev: Marked as regression because -10.963400652506964 (T) = (0 -325373.3502564252) / Math.Sqrt((38757820.72321942 / (33)) + (113084430.23404147 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.0890275676140183 = (298774.20914999885 - 325373.3502564252) / 298774.20914999885 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
kunalspathak commented 1 year ago

Some of the benchmarks have regressed like those in https://github.com/dotnet/runtime/issues/79973