search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/x64: 2 Regressions on 9/21/2022 6:35:49 PM #8703

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 49751b22745881ce98f84dddf31557c33546de36
Compare 25454874f2f82fb0741d04eab3262860c21a87e8
Diff Diff

Regressions in System.Collections.CopyTo<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadOnlySpan - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Collections.CopyTo(Int32).ReadOnlySpan(Size%3a%202048).html>) 129.86 ns 165.42 ns 1.27 0.29 False
[Span - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Collections.CopyTo(Int32).Span(Size%3a%202048).html>) 131.61 ns 163.95 ns 1.25 0.28 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CopyTo&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Collections.CopyTo<Int32>.ReadOnlySpan(Size: 2048) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 165.41691317786092 > 137.92296755207616. IsChangePoint: Marked as a change because one of 8/27/2022 12:34:40 AM, 9/21/2022 2:58:31 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsRegressionStdDev: Marked as regression because -7.013086397382641 (T) = (0 -164.39126503198406) / Math.Sqrt((262.20946739920527 / (31)) + (46.152392078876375 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.15745074166024178 = (142.02873531894932 - 164.39126503198406) / 142.02873531894932 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CopyTo<Int32>.Span(Size: 2048) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 163.9454004917322 > 138.11299833042682. IsChangePoint: Marked as a change because one of 7/22/2022 2:06:02 PM, 8/27/2022 12:34:40 AM, 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsRegressionStdDev: Marked as regression because -10.971176173790258 (T) = (0 -166.20665691688208) / Math.Sqrt((206.13527949604855 / (35)) + (2.4934669015371558 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.1928361608717414 = (139.33737286721416 - 166.20665691688208) / 139.33737286721416 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 49751b22745881ce98f84dddf31557c33546de36
Compare 25454874f2f82fb0741d04eab3262860c21a87e8
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[NewDelegateClassEmptyStaticFn - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/PerfLabTests.LowLevelPerf.NewDelegateClassEmptyStaticFn.html>) 473.32 μs 729.05 μs 1.54 0.61 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### PerfLabTests.LowLevelPerf.NewDelegateClassEmptyStaticFn ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 729.0546428571429 > 489.2256948728355. IsChangePoint: Marked as a change because one of 8/13/2022 7:28:42 AM, 9/21/2022 2:58:31 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM. IsRegressionStdDev: Marked as regression because -4.581825567314958 (T) = (0 -608786.7821288811) / Math.Sqrt((5120694547.223497 / (31)) + (12798094456.159945 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.23499095494882824 = (492948.3731758231 - 608786.7821288811) / 492948.3731758231 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)