search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 6/27/2022 5:19:23 PM #6581

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 988f570dc4ddd79a166ca8cbc184b2938ed2fefc
Compare 601ef262b5eca9a48a6207d4aa7f304c0827fdc8
Diff Diff

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TrailingZeroCount_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_uint.html>) 534.16 ns 631.31 ns 1.18 0.00 False
[LeadingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong.html>) 532.18 ns 630.77 ns 1.19 0.00 False
[TrailingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong.html>) 533.47 ns 631.33 ns 1.18 0.00 False
[LeadingZeroCount_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint.html>) 532.89 ns 631.20 ns 1.18 0.00 False

graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 631.3140193376631 > 560.1841846934802. IsChangePoint: Marked as a change because one of 6/26/2022 6:11:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -678.2716806874066 (T) = (0 -631.3217571667014) / Math.Sqrt((0.2698119403127713 / (13)) + (0.0009238819998096573 / (40))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (40) - 2, .025) and -0.183242952156297 = (533.5520959716723 - 631.3217571667014) / 533.5520959716723 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 630.7731483358333 > 559.4877124461453. IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -451.0364466959665 (T) = (0 -631.1316492592645) / Math.Sqrt((0.5997892143185513 / (13)) + (0.03590007288958501 / (38))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (38) - 2, .025) and -0.1835265215025134 = (533.2636301703055 - 631.1316492592645) / 533.2636301703055 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 631.3325552348496 > 560.2341092883905. IsChangePoint: Marked as a change because one of 6/26/2022 6:11:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -1429.0298962584402 (T) = (0 -631.3238264269291) / Math.Sqrt((0.055434850324261736 / (12)) + (0.0026633890475223584 / (39))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (39) - 2, .025) and -0.1834044070458352 = (533.4810506603741 - 631.3238264269291) / 533.4810506603741 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 631.2001866731597 > 559.9831246439804. IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -309.33770492837533 (T) = (0 -631.1529154230166) / Math.Sqrt((1.2897399177660482 / (13)) + (0.024011306944770675 / (37))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (37) - 2, .025) and -0.1832629626023808 = (533.4003812938636 - 631.1529154230166) / 533.4003812938636 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 988f570dc4ddd79a166ca8cbc184b2938ed2fefc
Compare 601ef262b5eca9a48a6207d4aa7f304c0827fdc8
Diff Diff

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Compare_Same_Upper - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20OrdinalIgnoreCase)).html>) 1.81 μs 2.02 μs 1.11 0.04 False
[Compare_Same - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_Same(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20Ordinal)).html>) 49.40 ns 77.38 ns 1.57 0.06 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20Ordinal)).html>) 10.33 ns 34.94 ns 3.38 0.04 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(%2c%20IgnoreCase)).html>) 38.09 ns 65.73 ns 1.73 0.23 False
[Compare_Same_Upper - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20Ordinal)).html>) 14.22 ns 40.27 ns 2.83 0.06 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(pl-PL%2c%20None)).html>) 38.81 ns 60.59 ns 1.56 0.18 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20OrdinalIgnoreCase)).html>) 9.73 ns 29.89 ns 3.07 0.04 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(%2c%20None)).html>) 39.20 ns 64.32 ns 1.64 0.25 False
[Compare_DifferentFirstChar - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count%3a%201024%2c%20Options%3a%20(en-US%2c%20IgnoreNonSpace)).html>) 37.94 ns 60.64 ns 1.60 0.20 False

graph graph graph graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringEquality*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.0195216486440097 > 1.8909675887604789. IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/11/2022 7:02:26 PM, 6/26/2022 6:11:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -23.468241582741786 (T) = (0 -2211.884059701522) / Math.Sqrt((33.4388265374827 / (12)) + (11945.317529858245 / (39))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (39) - 2, .025) and -0.22930337192366074 = (1799.298781911158 - 2211.884059701522) / 1799.298781911158 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, Ordinal)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 77.37534776006848 > 52.87466914379532. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -88.67873548410091 (T) = (0 -76.83054926226555) / Math.Sqrt((1.1573242926484686 / (40)) + (0.3071373319814177 / (5))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (5) - 2, .025) and -0.5312915890389797 = (50.17368985255348 - 76.83054926226555) / 50.17368985255348 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, Ordinal)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 34.94012592441773 > 10.849129475605542. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -480.98650590964587 (T) = (0 -34.89601926325484) / Math.Sqrt((0.00046897451389723806 / (41)) + (0.018148477076157767 / (7))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (7) - 2, .025) and -2.3712060632477145 = (10.351197348534996 - 34.89601926325484) / 10.351197348534996 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, IgnoreCase)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 65.73176879310567 > 40.242816332087315. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -29.03183265762772 (T) = (0 -62.693675414697516) / Math.Sqrt((1.4791466299864098 / (38)) + (4.465688618598093 / (7))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (7) - 2, .025) and -0.6154670414841715 = (38.808390270283205 - 62.693675414697516) / 38.808390270283205 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, Ordinal)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 40.27209334914114 > 15.00140324284727. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -97.24590521368253 (T) = (0 -38.955393675416914) / Math.Sqrt((0.07194692774639135 / (46)) + (0.43786507247455636 / (7))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (7) - 2, .025) and -1.718150142411213 = (14.331582743571484 - 38.955393675416914) / 14.331582743571484 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (pl-PL, None)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 60.58796788982129 > 41.2094875156975. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -47.542644755306654 (T) = (0 -61.647827963219676) / Math.Sqrt((1.8982151622909413 / (44)) + (1.3439641106135465 / (7))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (7) - 2, .025) and -0.597342213101642 = (38.59400162192853 - 61.647827963219676) / 38.59400162192853 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, OrdinalIgnoreCase)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 29.890049998114574 > 10.215100845063143. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -161.75552872299178 (T) = (0 -29.984979473881168) / Math.Sqrt((0.0344117776679848 / (38)) + (0.07621296776983906 / (5))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (5) - 2, .025) and -2.179802327019338 = (9.429824998583573 - 29.984979473881168) / 9.429824998583573 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, None)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 64.32119422306181 > 41.45880593439457. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -29.009334036798194 (T) = (0 -62.411884700758144) / Math.Sqrt((3.1450348602688045 / (41)) + (4.066294685852578 / (7))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (7) - 2, .025) and -0.6049406908399291 = (38.8873464651803 - 62.411884700758144) / 38.8873464651803 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreNonSpace)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 60.643547668760036 > 41.948209850867045. IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -57.182024600186324 (T) = (0 -61.74783064050785) / Math.Sqrt((2.622931215640502 / (42)) + (0.592170926124694 / (6))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (6) - 2, .025) and -0.5917153921243563 = (38.79326099755632 - 61.74783064050785) / 38.79326099755632 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare 601ef262b5eca9a48a6207d4aa7f304c0827fdc8
Diff Diff

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IFooObjIsIFooInterAlia - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia.html>) 377.54 μs 420.50 μs 1.11 0.12 False
[IFooObjIsIFoo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo.html>) 501.42 μs 551.45 μs 1.10 0.04 False
[IFooObjIsDescendantOfIFoo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsDescendantOfIFoo.html>) 501.66 μs 548.32 μs 1.09 0.03 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 420.49608552631577 > 396.648828125. IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 5/10/2022 5:59:50 PM, 5/31/2022 6:06:22 PM, 6/27/2022 8:54:23 AM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -91.45201322214768 (T) = (0 -418824.2214195327) / Math.Sqrt((1018450.8266521781 / (12)) + (4557894.959050034 / (39))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (39) - 2, .025) and -0.10873941149286898 = (377748.1138292038 - 418824.2214195327) / 377748.1138292038 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 551.446882183908 > 526.5905030241936. IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/10/2022 3:36:59 PM, 6/27/2022 8:54:23 AM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -156.27100868123304 (T) = (0 -552134.779376708) / Math.Sqrt((896564.0264295251 / (13)) + (1250898.26773376 / (35))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (35) - 2, .025) and -0.1008173352238772 = (501568.0274188436 - 552134.779376708) / 501568.0274188436 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsDescendantOfIFoo ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 548.3214747536946 > 526.8928716195913. IsChangePoint: Marked as a change because one of 6/26/2022 6:11:06 PM, 7/5/2022 10:15:28 AM falls between 6/26/2022 6:11:06 PM and 7/5/2022 10:15:28 AM. IsRegressionStdDev: Marked as regression because -108.01819003208878 (T) = (0 -549677.3734081823) / Math.Sqrt((1595446.726914541 / (11)) + (1786673.1888377706 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.09457662041837722 = (502182.6367879852 - 549677.3734081823) / 502182.6367879852 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
kunalspathak commented 2 years ago

For some benchmarks, the performance came back while for some, it falls to https://github.com/dotnet/runtime/compare/52f4e7c567bbf1d239c0330a9bae3e929cb40229...601ef262b5eca9a48a6207d4aa7f304c0827fdc8 which might point to the PGO.