search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.25k stars 4.73k forks source link

Regressions in System.Tests.Perf_UInt64 #71771

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SortedList - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.ContainsKeyFalse(String%2c%20String).SortedList(Size%3a%20512).html>) 348.38 μs 384.99 μs 1.11 0.03 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyFalse<String, String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 384.9944918699187 > 366.0416569444444. IsChangePoint: Marked as a change because one of 6/3/2022 8:17:54 AM, 6/6/2022 3:15:19 PM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -18.592276578561822 (T) = (0 -386239.64188622445) / Math.Sqrt((827784.6548322291 / (25)) + (73255526.55193321 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.10803726811218585 = (348580.00989829405 - 386239.64188622445) / 348580.00989829405 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryParseHex - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Tests.Perf_UInt64.TryParseHex(value%3a%20%220%22).html>) 4.52 ns 5.85 ns 1.30 0.35 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_UInt64.TryParseHex(value: "0") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.851650109031447 > 5.099529018869719. IsChangePoint: Marked as a change because one of 3/7/2022 12:49:56 AM, 3/7/2022 10:45:01 PM, 3/12/2022 3:27:04 PM, 3/13/2022 11:50:20 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -22.045017915913398 (T) = (0 -5.800006004212765) / Math.Sqrt((0.03388441533165363 / (24)) + (0.012577560483055143 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.2115580874843213 = (4.7872289939113815 - 5.800006004212765) / 4.7872289939113815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IsDefinedMethodOverrideMiss - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Reflection.Attributes.IsDefinedMethodOverrideMiss.html>) 660.32 ns 694.28 ns 1.05 0.18 False
[IsDefinedClassMissInherit - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Reflection.Attributes.IsDefinedClassMissInherit.html>) 1.11 μs 1.21 μs 1.09 0.13 False
[IsDefinedMethodOverrideHit - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Reflection.Attributes.IsDefinedMethodOverrideHit.html>) 654.04 ns 705.29 ns 1.08 0.15 False
[IsDefinedMethodBaseHitInherit - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Reflection.Attributes.IsDefinedMethodBaseHitInherit.html>) 660.85 ns 701.58 ns 1.06 0.16 False
[IsDefinedClassHit - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Reflection.Attributes.IsDefinedClassHit.html>) 606.11 ns 673.82 ns 1.11 0.19 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Reflection.Attributes.IsDefinedMethodOverrideMiss ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 694.281147282633 > 668.6631138733229. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -8.451246977120563 (T) = (0 -708.0806406154733) / Math.Sqrt((495.03377189893916 / (25)) + (567.5766624145348 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.09351064462495992 = (647.5297191627457 - 708.0806406154733) / 647.5297191627457 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedClassMissInherit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.2112155248820418 > 1.13694964638556. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -7.745527387390561 (T) = (0 -1213.9608926620335) / Math.Sqrt((1954.7879162499878 / (24)) + (1039.2474975435828 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08140105517635117 = (1122.581568467274 - 1213.9608926620335) / 1122.581568467274 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedMethodOverrideHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 705.2855175216763 > 693.5954236978833. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.1016749220948405 (T) = (0 -709.1377152585584) / Math.Sqrt((647.7044138314708 / (25)) + (994.2426132288017 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.08402020507970037 = (654.1738907960857 - 709.1377152585584) / 654.1738907960857 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedMethodBaseHitInherit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 701.575127475433 > 696.2912043948237. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.276517735042958 (T) = (0 -709.118067717889) / Math.Sqrt((788.1459480483514 / (24)) + (809.2232784828515 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08468007639528252 = (653.7578067023238 - 709.118067717889) / 653.7578067023238 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedClassHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 673.8237605760315 > 640.9646320332743. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.410882182847598 (T) = (0 -669.5355188242203) / Math.Sqrt((867.0339451611512 / (17)) + (709.075798026562 / (26))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (26) - 2, .025) and -0.09255424522176141 = (612.8167290113098 - 669.5355188242203) / 612.8167290113098 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

dakersnar commented 2 years ago

@stephentoub regression in that second benchmark, we think it's caused by https://github.com/dotnet/runtime/pull/71146. Based on the PR this might be an expected regression.

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-globalization See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [b92de6bf0351280cd36221f3232b2964a4e61e88](https://github.com/dotnet/runtime/commit/b92de6bf0351280cd36221f3232b2964a4e61e88) Compare | [d4a9ade2dfbee1ef532e7793ea9c330c51b5c028](https://github.com/dotnet/runtime/commit/d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) Diff | [Diff](https://github.com/dotnet/runtime/compare/b92de6bf0351280cd36221f3232b2964a4e61e88...d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) ### Regressions in System.Collections.ContainsKeyFalse<String, String> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [SortedList - Duration of single invocation]() | 348.38 μs | 384.99 μs | 1.11 | 0.03 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse<String, String>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyFalse<String, String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 384.9944918699187 > 366.0416569444444. IsChangePoint: Marked as a change because one of 6/3/2022 8:17:54 AM, 6/6/2022 3:15:19 PM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -18.592276578561822 (T) = (0 -386239.64188622445) / Math.Sqrt((827784.6548322291 / (25)) + (73255526.55193321 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.10803726811218585 = (348580.00989829405 - 386239.64188622445) / 348580.00989829405 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [b92de6bf0351280cd36221f3232b2964a4e61e88](https://github.com/dotnet/runtime/commit/b92de6bf0351280cd36221f3232b2964a4e61e88) Compare | [d4a9ade2dfbee1ef532e7793ea9c330c51b5c028](https://github.com/dotnet/runtime/commit/d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) Diff | [Diff](https://github.com/dotnet/runtime/compare/b92de6bf0351280cd36221f3232b2964a4e61e88...d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) ### Regressions in System.Tests.Perf_UInt64 Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [TryParseHex - Duration of single invocation]() | 4.52 ns | 5.85 ns | 1.30 | 0.35 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_UInt64.TryParseHex(value: "0") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.851650109031447 > 5.099529018869719. IsChangePoint: Marked as a change because one of 3/7/2022 12:49:56 AM, 3/7/2022 10:45:01 PM, 3/12/2022 3:27:04 PM, 3/13/2022 11:50:20 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -22.045017915913398 (T) = (0 -5.800006004212765) / Math.Sqrt((0.03388441533165363 / (24)) + (0.012577560483055143 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.2115580874843213 = (4.7872289939113815 - 5.800006004212765) / 4.7872289939113815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [b92de6bf0351280cd36221f3232b2964a4e61e88](https://github.com/dotnet/runtime/commit/b92de6bf0351280cd36221f3232b2964a4e61e88) Compare | [d4a9ade2dfbee1ef532e7793ea9c330c51b5c028](https://github.com/dotnet/runtime/commit/d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) Diff | [Diff](https://github.com/dotnet/runtime/compare/b92de6bf0351280cd36221f3232b2964a4e61e88...d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) ### Regressions in System.Reflection.Attributes Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IsDefinedMethodOverrideMiss - Duration of single invocation]() | 660.32 ns | 694.28 ns | 1.05 | 0.18 | False | | | [IsDefinedClassMissInherit - Duration of single invocation]() | 1.11 μs | 1.21 μs | 1.09 | 0.13 | False | | | [IsDefinedMethodOverrideHit - Duration of single invocation]() | 654.04 ns | 705.29 ns | 1.08 | 0.15 | False | | | [IsDefinedMethodBaseHitInherit - Duration of single invocation]() | 660.85 ns | 701.58 ns | 1.06 | 0.16 | False | | | [IsDefinedClassHit - Duration of single invocation]() | 606.11 ns | 673.82 ns | 1.11 | 0.19 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Reflection.Attributes.IsDefinedMethodOverrideMiss ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 694.281147282633 > 668.6631138733229. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -8.451246977120563 (T) = (0 -708.0806406154733) / Math.Sqrt((495.03377189893916 / (25)) + (567.5766624145348 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.09351064462495992 = (647.5297191627457 - 708.0806406154733) / 647.5297191627457 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedClassMissInherit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.2112155248820418 > 1.13694964638556. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -7.745527387390561 (T) = (0 -1213.9608926620335) / Math.Sqrt((1954.7879162499878 / (24)) + (1039.2474975435828 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08140105517635117 = (1122.581568467274 - 1213.9608926620335) / 1122.581568467274 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedMethodOverrideHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 705.2855175216763 > 693.5954236978833. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.1016749220948405 (T) = (0 -709.1377152585584) / Math.Sqrt((647.7044138314708 / (25)) + (994.2426132288017 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.08402020507970037 = (654.1738907960857 - 709.1377152585584) / 654.1738907960857 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedMethodBaseHitInherit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 701.575127475433 > 696.2912043948237. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.276517735042958 (T) = (0 -709.118067717889) / Math.Sqrt((788.1459480483514 / (24)) + (809.2232784828515 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08468007639528252 = (653.7578067023238 - 709.118067717889) / 653.7578067023238 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedClassHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 673.8237605760315 > 640.9646320332743. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.410882182847598 (T) = (0 -669.5355188242203) / Math.Sqrt((867.0339451611512 / (17)) + (709.075798026562 / (26))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (26) - 2, .025) and -0.09255424522176141 = (612.8167290113098 - 669.5355188242203) / 612.8167290113098 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: dakersnar
Labels: `area-System.Globalization`, `tenet-performance`, `tenet-performance-benchmarks`, `untriaged`, `refs/heads/main`, `RunKind=micro`, `Windows 10.0.19041`, `Regression`, `CoreClr`, `arm64`
Milestone: -
stephentoub commented 2 years ago

@stephentoub regression in that second benchmark, we think it's caused by https://github.com/dotnet/runtime/pull/71146. Based on the PR this might be an expected regression.

I'd be surprised if it directly contributed... the method that changed shouldn't be invoked in that benchmark (even though it's in the right area of the code)... it's a non-inlineable method that's only used if there are trailing nulls, which this test doesn't have.

I'll take a look, though.

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-runtime See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [b92de6bf0351280cd36221f3232b2964a4e61e88](https://github.com/dotnet/runtime/commit/b92de6bf0351280cd36221f3232b2964a4e61e88) Compare | [d4a9ade2dfbee1ef532e7793ea9c330c51b5c028](https://github.com/dotnet/runtime/commit/d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) Diff | [Diff](https://github.com/dotnet/runtime/compare/b92de6bf0351280cd36221f3232b2964a4e61e88...d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) ### Regressions in System.Collections.ContainsKeyFalse<String, String> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [SortedList - Duration of single invocation]() | 348.38 μs | 384.99 μs | 1.11 | 0.03 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse<String, String>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsKeyFalse<String, String>.SortedList(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 384.9944918699187 > 366.0416569444444. IsChangePoint: Marked as a change because one of 6/3/2022 8:17:54 AM, 6/6/2022 3:15:19 PM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -18.592276578561822 (T) = (0 -386239.64188622445) / Math.Sqrt((827784.6548322291 / (25)) + (73255526.55193321 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.10803726811218585 = (348580.00989829405 - 386239.64188622445) / 348580.00989829405 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [b92de6bf0351280cd36221f3232b2964a4e61e88](https://github.com/dotnet/runtime/commit/b92de6bf0351280cd36221f3232b2964a4e61e88) Compare | [d4a9ade2dfbee1ef532e7793ea9c330c51b5c028](https://github.com/dotnet/runtime/commit/d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) Diff | [Diff](https://github.com/dotnet/runtime/compare/b92de6bf0351280cd36221f3232b2964a4e61e88...d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) ### Regressions in System.Tests.Perf_UInt64 Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [TryParseHex - Duration of single invocation]() | 4.52 ns | 5.85 ns | 1.30 | 0.35 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_UInt64.TryParseHex(value: "0") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.851650109031447 > 5.099529018869719. IsChangePoint: Marked as a change because one of 3/7/2022 12:49:56 AM, 3/7/2022 10:45:01 PM, 3/12/2022 3:27:04 PM, 3/13/2022 11:50:20 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -22.045017915913398 (T) = (0 -5.800006004212765) / Math.Sqrt((0.03388441533165363 / (24)) + (0.012577560483055143 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.2115580874843213 = (4.7872289939113815 - 5.800006004212765) / 4.7872289939113815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.19041 Baseline | [b92de6bf0351280cd36221f3232b2964a4e61e88](https://github.com/dotnet/runtime/commit/b92de6bf0351280cd36221f3232b2964a4e61e88) Compare | [d4a9ade2dfbee1ef532e7793ea9c330c51b5c028](https://github.com/dotnet/runtime/commit/d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) Diff | [Diff](https://github.com/dotnet/runtime/compare/b92de6bf0351280cd36221f3232b2964a4e61e88...d4a9ade2dfbee1ef532e7793ea9c330c51b5c028) ### Regressions in System.Reflection.Attributes Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IsDefinedMethodOverrideMiss - Duration of single invocation]() | 660.32 ns | 694.28 ns | 1.05 | 0.18 | False | | | [IsDefinedClassMissInherit - Duration of single invocation]() | 1.11 μs | 1.21 μs | 1.09 | 0.13 | False | | | [IsDefinedMethodOverrideHit - Duration of single invocation]() | 654.04 ns | 705.29 ns | 1.08 | 0.15 | False | | | [IsDefinedMethodBaseHitInherit - Duration of single invocation]() | 660.85 ns | 701.58 ns | 1.06 | 0.16 | False | | | [IsDefinedClassHit - Duration of single invocation]() | 606.11 ns | 673.82 ns | 1.11 | 0.19 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Reflection.Attributes.IsDefinedMethodOverrideMiss ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 694.281147282633 > 668.6631138733229. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -8.451246977120563 (T) = (0 -708.0806406154733) / Math.Sqrt((495.03377189893916 / (25)) + (567.5766624145348 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.09351064462495992 = (647.5297191627457 - 708.0806406154733) / 647.5297191627457 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedClassMissInherit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.2112155248820418 > 1.13694964638556. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -7.745527387390561 (T) = (0 -1213.9608926620335) / Math.Sqrt((1954.7879162499878 / (24)) + (1039.2474975435828 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08140105517635117 = (1122.581568467274 - 1213.9608926620335) / 1122.581568467274 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedMethodOverrideHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 705.2855175216763 > 693.5954236978833. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.1016749220948405 (T) = (0 -709.1377152585584) / Math.Sqrt((647.7044138314708 / (25)) + (994.2426132288017 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.08402020507970037 = (654.1738907960857 - 709.1377152585584) / 654.1738907960857 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedMethodBaseHitInherit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 701.575127475433 > 696.2912043948237. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.276517735042958 (T) = (0 -709.118067717889) / Math.Sqrt((788.1459480483514 / (24)) + (809.2232784828515 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08468007639528252 = (653.7578067023238 - 709.118067717889) / 653.7578067023238 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Reflection.Attributes.IsDefinedClassHit ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 673.8237605760315 > 640.9646320332743. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -6.410882182847598 (T) = (0 -669.5355188242203) / Math.Sqrt((867.0339451611512 / (17)) + (709.075798026562 / (26))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (26) - 2, .025) and -0.09255424522176141 = (612.8167290113098 - 669.5355188242203) / 612.8167290113098 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: dakersnar
Labels: `area-System.Runtime`, `tenet-performance`, `tenet-performance-benchmarks`, `untriaged`, `refs/heads/main`, `RunKind=micro`, `Windows 10.0.19041`, `Regression`, `CoreClr`, `arm64`
Milestone: -
dakersnar commented 2 years ago

Another similar regression https://github.com/dotnet/perf-autofiling-issues/issues/6494

dakersnar commented 2 years ago

We have no reasonable steps to fix this regression, and it is on the nanosecond scale, so the impact is low. Closing this.

mrsharm commented 2 years ago

A similar regression for showed up for the Windows x64 configuration in our August Performance report and has been characterized as "by design":

image

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615") | Result | Ratio | Alloc Delta | Operating System | Bit | Processor Name | | ------ | -----:| -----------:| --------------------- | ----- | ----------------------------------------------- | | Same | 1.00 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | Same | 0.98 | +0 | Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz | | Slower | 0.85 | +0 | macOS Monterey 12.3 | Arm64 | Apple M1 Max | | Slower | 0.85 | +0 | Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | Same | 0.92 | +0 | Windows 10 | X64 | Intel Core i7-6700 CPU 3.40GHz (Skylake) | | Faster | 1.12 | +0 | Windows 10 | X64 | Intel Core i7-6700 CPU 3.40GHz (Skylake) | | Same | 1.10 | +0 | Windows 10 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | | Slower | 0.85 | +0 | Windows 10 | X64 | Intel Core i9-10900K CPU 3.70GHz | | Same | 1.00 | +0 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | Same | 0.96 | +0 | Windows 11 | X64 | AMD Ryzen 9 3950X | | Slower | 0.86 | +0 | Windows 11 | X64 | AMD Ryzen 9 5900X | | Same | 1.08 | +0 | Windows 11 | X64 | AMD Ryzen 9 5950X | | Same | 1.07 | +0 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | Same | 0.91 | +0 | Windows 11 | X64 | Intel Core i9-10900K CPU 3.70GHz | | Same | 0.91 | +0 | Windows 11 | X64 | 11th Gen Intel Core i9-11900H 2.50GHz | | Slower | 0.90 | +0 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | | Same | 1.01 | +0 | ubuntu 18.04 | X64 | Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) | | Slower | 0.78 | +0 | ubuntu 18.04 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | | Slower | 0.80 | +0 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | | Slower | 0.87 | +0 | ubuntu 20.04 | X64 | Intel Core i9-10900K CPU 3.70GHz | | Same | 0.94 | +0 | Windows 10 | X86 | Intel Xeon CPU E5-1650 v4 3.60GHz | | Same | 0.99 | +0 | Windows 10 | X86 | Intel Core i7-6700 CPU 3.40GHz (Skylake) | | Same | 0.96 | +0 | Windows 11 | X86 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | | Slower | 0.85 | +0 | macOS Big Sur 11.6.8 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | | Slower | 0.87 | +0 | macOS Monterey 12.3.1 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | | Slower | 0.80 | +0 | macOS Monterey 12.4 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) |