DrewScoggins / performance-2

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
0 stars 0 forks source link

[Perf] Windows/arm64: 1 Regression on 10/2/2022 4:08:26 PM #8187

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.25094
Baseline 319391eb90065ec2aca29c6de0046e47a1286016
Compare 144a33a01251fce466d7be9b7cafff02f05ae6cc
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Dequeue_And_Enqueue - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.25094/System.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).Dequeue_And_Enqueue(Size%3a%20100).html>) 12.52 μs 14.66 μs 1.17 0.62 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'
### Related Issues #### Regressions * [[Perf] Linux/arm64: 2 Regressions on 10/2/2022 4:08:26 PM]() * [[Perf] Linux/x64: 4 Regressions on 10/2/2022 4:08:26 PM]() * [[Perf] Linux/x64: 1 Regression on 10/2/2022 4:08:26 PM]() #### Improvements ### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.Dequeue_And_Enqueue(Size: 100) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 14.65921994274809 > 11.766507442740314. IsChangePoint: Marked as a change because one of 10/2/2022 5:38:34 AM, 10/6/2022 12:41:43 AM falls between 9/27/2022 9:53:04 AM and 10/6/2022 12:41:43 AM. IsRegressionStdDev: Marked as regression because -17.35717414230426 (T) = (0 -16712.385048239965) / Math.Sqrt((803929.8360672664 / (35)) + (1355235.77981829 / (17))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (17) - 2, .025) and -0.4988347070165136 = (11150.252239292344 - 16712.385048239965) / 11150.252239292344 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
DrewScoggins commented 2 years ago

https://github.com/dotnet/runtime/pull/76155