dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 1 Regression on 10/11/2022 6:23:40 AM #9051

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline d6fdb3ab48ba885e5386a529ce53e143d56ab963
Compare d6c8ba1f7806c03b90047bcf8fa4eabdf0e143a3
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryFormat - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_UInt64.TryFormat(value%3a%200).html>) 242.28 ns 267.77 ns 1.11 0.23 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_UInt64.TryFormat(value: 0) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 267.77236034758965 > 248.3236066649993. IsChangePoint: Marked as a change because one of 7/28/2022 3:42:13 AM, 10/11/2022 2:16:14 AM, 10/18/2022 9:26:37 AM falls between 10/7/2022 9:25:17 PM and 10/18/2022 9:26:37 AM. IsRegressionStdDev: Marked as regression because -12.878283726638552 (T) = (0 -271.3657644737842) / Math.Sqrt((17.600888089904377 / (17)) + (39.02524546278677 / (7))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (7) - 2, .025) and -0.13897270434517453 = (238.25484442122743 - 271.3657644737842) / 238.25484442122743 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

looks like https://github.com/dotnet/runtime/pull/76726