dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 7/15/2022 7:20:55 PM #6906

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 68e13b68bc20a36a0cfd715d88cf04e4fb9b510b
Compare 00d9488a70ad6e4e8d7aa6e0325d0deba0a14957
Diff Diff

Regressions in System.IO.Compression.Brotli

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.IO.Compression.Brotli.Compress(level%3a%20Optimal%2c%20file%3a%20%22TestDocument.pdf%22).html>) 13.59 μs 423.38 μs 31.16 0.38 False
[Compress - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.18362/System.IO.Compression.Brotli.Compress(level%3a%20Optimal%2c%20file%3a%20%22alice29.txt%22).html>) 17.54 μs 2.59 ms 147.78 0.51 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Compression.Brotli*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 423.37709821428575 > 13.955767172445656. IsChangePoint: Marked as a change because one of 7/15/2022 5:00:56 PM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -215.66452603817729 (T) = (0 -427496.7062783447) / Math.Sqrt((757370.4008776997 / (72)) + (51529833.63553704 / (14))) is less than -1.9886096669751727 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (72) + (14) - 2, .025) and -31.5089580427055 = (13150.120213535058 - 427496.7062783447) / 13150.120213535058 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.591364940476191 > 19.55052447696742. IsChangePoint: Marked as a change because one of 7/15/2022 5:00:56 PM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM. IsRegressionStdDev: Marked as regression because -1414.9218726431786 (T) = (0 -2592780.565476191) / Math.Sqrt((2789479.294336968 / (72)) + (45799989.930207685 / (14))) is less than -1.9886096669751727 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (72) + (14) - 2, .025) and -139.22992053030035 = (18489.496076666128 - 2592780.565476191) / 18489.496076666128 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
DrewScoggins commented 2 years ago

https://github.com/dotnet/runtime/issues/73391#issuecomment-1205619431