dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] ubuntu 18.04/x64 : Regression on 8/21/2022 7:03:28 PM #8150

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 729844af1a22c02e6afe53d8053dbd5ec9c5db7e
Compare 05881294b9cef6212494f850410ab01cb69e9a99
Diff Diff

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadAllBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_File.ReadAllBytes(size%3a%20104857600).html>) 0.00 ns 16.44 ms 0.10 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_File*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_File.ReadAllBytes(size: 104857600) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.440551106666664 > 0. IsChangePoint: Marked as a change because one of 8/19/2022 10:17:27 PM, 8/21/2022 6:52:55 AM, 8/30/2022 4:40:43 AM falls between 8/21/2022 6:52:55 AM and 8/30/2022 4:40:43 AM. IsRegressionStdDev: Marked as regression because -3.323415890284529 (T) = (0 -16192273.591717556) / Math.Sqrt((68569060578721.18 / (13)) + (553959442966.1699 / (48))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (48) - 2, .025) and -0.8935570310173376 = (8551246.847325243 - 16192273.591717556) / 8551246.847325243 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
naricc commented 2 years ago

Same with this @DrewScoggins What does the 0 mean?