dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 8/3/2022 8:16:08 AM #7139

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 9a3626e16efb4fb1ae75d806bf66f0769bd842c1
Compare ed561a2d2f88d9a585b8db95dda03d8c0de4ae4e
Diff Diff

Regressions in System.Collections.IterateForEach<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableQueue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 20.04/System.Collections.IterateForEach(Int32).ImmutableQueue(Size%3a%20512).html>) 4.17 μs 4.63 μs 1.11 0.04 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IterateForEach<Int32>.ImmutableQueue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.625698468228086 > 4.3906914773672066. IsChangePoint: Marked as a change because one of 5/24/2022 3:42:52 PM, 7/28/2022 12:19:57 AM, 8/3/2022 1:34:55 AM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM. IsRegressionStdDev: Marked as regression because -11.413131244469195 (T) = (0 -4628.167642803265) / Math.Sqrt((36971.948630060106 / (42)) + (5.295360822078732 / (3))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (3) - 2, .025) and -0.0790268139637435 = (4289.205405194663 - 4628.167642803265) / 4289.205405194663 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 2 years ago

@adamsitnik interesting one here, BDN update destabilized this benchmark

adamsitnik commented 2 years ago

@adamsitnik interesting one here, BDN update destabilized this benchmark

I double checked and the recent BDN update did not contain any Engine code changes. Just .NET 4.8.1 support, NativeAOT fixes and few minor bugs. We have added net6.0 target to the library, but without code changes (except of adding new attributes to few methods).