dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/x64: 1 Regression on 2/9/2023 1:20:08 AM #12917

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 43a60c8ed073a4c6134facadd01c9c1c2643e41a
Compare 3ff80e90e828bac0370c1930c9950c9650ae61b9
Diff Diff

Regressions in System.Collections.CtorDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ConcurrentDictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.Collections.CtorDefaultSize(String).ConcurrentDictionary.html>) 117.64 ns 130.07 ns 1.11 0.05 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorDefaultSize&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorDefaultSize<String>.ConcurrentDictionary ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 130.07322718515854 > 124.18865768927303. IsChangePoint: Marked as a change because one of 2/8/2023 3:55:57 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM. IsRegressionStdDev: Marked as regression because -19.94428441336426 (T) = (0 -128.43721690890428) / Math.Sqrt((6.182967266981806 / (36)) + (4.685716643902182 / (24))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (24) - 2, .025) and -0.10383855602879141 = (116.35507403453504 - 128.43721690890428) / 116.35507403453504 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
stephentoub commented 1 year ago

A small increase (this shows it at ~10ns) in the cost of constructing a ConcurrentDictionary<string, ...> is expected, as it's now doing a little bit more work to select a non-randomized comparer if possible (which in turn makes most subsequent operations on the dictionary much faster).