search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 2 Regressions on 9/23/2022 10:31:05 PM #8707

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 38ce8bb2422e14c142254a72f02382dfa5b644c7
Compare 901b0a1ee23c441e255acaf032adbf709f36d6a6
Diff Diff

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Fill - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Memory.Span(Char).Fill(Size%3a%20512).html>) 8.40 ns 15.35 ns 1.83 0.48 False
[StartsWith - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Memory.Span(Char).StartsWith(Size%3a%20512).html>) 8.83 ns 11.37 ns 1.29 0.05 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Char&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Memory.Span<Char>.Fill(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.350916749006378 > 8.836394884547513. IsChangePoint: Marked as a change because one of 9/23/2022 6:26:22 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsRegressionStdDev: Marked as regression because -4.016961106764327 (T) = (0 -13.229472718260377) / Math.Sqrt((3.658036405156422 / (41)) + (9.415211631187047 / (17))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (17) - 2, .025) and -0.3218581101243409 = (10.008239626427033 - 13.229472718260377) / 10.008239626427033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Char>.StartsWith(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.370417337257797 > 9.284449902595908. IsChangePoint: Marked as a change because one of 7/27/2022 9:27:06 AM, 8/23/2022 11:46:46 PM, 9/23/2022 1:16:35 AM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsRegressionStdDev: Marked as regression because -48.345708787311814 (T) = (0 -11.188298745717333) / Math.Sqrt((0.061674286148541126 / (37)) + (0.00738255032086823 / (20))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (20) - 2, .025) and -0.24220011268813518 = (9.006840871641629 - 11.188298745717333) / 9.006840871641629 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 38ce8bb2422e14c142254a72f02382dfa5b644c7
Compare 901b0a1ee23c441e255acaf032adbf709f36d6a6
Diff Diff

Improvements in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Collections.CreateAddAndClear(Int32).IDictionary(Size%3a%20512).html>) 8.41 μs 7.59 μs 0.90 0.20 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Collections.CreateAddAndClear<Int32>.IDictionary(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 7.593085368380813 < 7.920085696778731. IsChangePoint: Marked as a change because one of 9/8/2022 1:37:01 AM, 9/23/2022 6:26:22 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsImprovementStdDev: Marked as improvement because 25.206533457087563 (T) = (0 -7490.300856230199) / Math.Sqrt((109347.37773211986 / (299)) + (19974.738362545675 / (17))) is greater than 1.9675476980854363 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (17) - 2, .975) and 0.11667853515307706 = (8479.699808413743 - 7490.300856230199) / 8479.699808413743 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 38ce8bb2422e14c142254a72f02382dfa5b644c7
Compare 901b0a1ee23c441e255acaf032adbf709f36d6a6
Diff Diff

Improvements in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ConcurrentStack - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.CtorFromCollection(Int32).ConcurrentStack(Size%3a%20512).html>) 4.37 μs 3.88 μs 0.89 0.18 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Collections.CtorFromCollection<Int32>.ConcurrentStack(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 3.876296433878157 < 4.024083540228242. IsChangePoint: Marked as a change because one of 8/5/2022 10:45:33 AM, 9/13/2022 12:23:36 PM, 9/16/2022 6:13:21 PM, 9/23/2022 6:26:22 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsImprovementStdDev: Marked as improvement because 9.260564310429492 (T) = (0 -3943.375355020465) / Math.Sqrt((38262.335875316116 / (299)) + (13674.855591393189 / (17))) is greater than 1.9675476980854363 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (17) - 2, .975) and 0.06690945457112067 = (4226.144369738479 - 3943.375355020465) / 4226.144369738479 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)