dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 1 Regression on 12/9/2022 8:25:19 AM #10698

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 41ae1aebcce81dbe742bad16533b70d0f46ef7c8
Compare 9333e468a9bc61950f3c7b25d52338dfd956ce78
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[DeserializeFromString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Serialization.Tests.ReadJson(SimpleStructWithProperties).DeserializeFromString(Mode%3a%20Reflection).html>) 3.17 μs 3.46 μs 1.09 0.16 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;SimpleStructWithProperties&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>.DeserializeFromString(Mode: Reflection) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.461238030973451 > 3.2572789959888455. IsChangePoint: Marked as a change because one of 12/9/2022 1:45:57 AM, 12/13/2022 5:38:37 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 5:38:37 AM. IsRegressionStdDev: Marked as regression because -10.347639815077581 (T) = (0 -3346.025689010052) / Math.Sqrt((4430.590692428418 / (38)) + (9476.468334611096 / (19))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (19) - 2, .025) and -0.08308814193938097 = (3089.338309085951 - 3346.025689010052) / 3089.338309085951 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

Probably tfm change https://github.com/dotnet/runtime/pull/79254