search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.95k stars 4.65k forks source link

Regressions in System.Threading.Tests.Perf_Timer #87826

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 19712d9cbeb086bf196c43b8b502e463702fc910
Compare 91cc34c9d1d884eaf71e1f33b86094071d5f1404
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tests.Perf_Timer

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SynchronousContention - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04/System.Threading.Tests.Perf_Timer.SynchronousContention.html>) 1.48 secs 2.02 secs 1.36 0.29 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_Timer*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Threading.Tests.Perf_Timer.SynchronousContention ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.0187393852666666 > 1.5564687483725. IsChangePoint: Marked as a change because one of 6/12/2023 3:15:58 AM, 6/19/2023 11:51:49 PM falls between 6/10/2023 7:23:52 PM and 6/19/2023 11:51:49 PM. IsRegressionStdDev: Marked as regression because -14.06732025634985 (T) = (0 -2104758606.1809525) / Math.Sqrt((2732912057190156 / (17)) + (47891860143521784 / (28))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (28) - 2, .025) and -0.40669280758699733 = (1496246085.022215 - 2104758606.1809525) / 1496246085.022215 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 1 year ago

Regressed via https://github.com/dotnet/runtime/pull/87218 cc @dotnet/gc

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | x64 OS | ubuntu 18.04 Queue | TigerUbuntu Baseline | [19712d9cbeb086bf196c43b8b502e463702fc910](https://github.com/dotnet/runtime/commit/19712d9cbeb086bf196c43b8b502e463702fc910) Compare | [91cc34c9d1d884eaf71e1f33b86094071d5f1404](https://github.com/dotnet/runtime/commit/91cc34c9d1d884eaf71e1f33b86094071d5f1404) Diff | [Diff](https://github.com/dotnet/runtime/compare/19712d9cbeb086bf196c43b8b502e463702fc910...91cc34c9d1d884eaf71e1f33b86094071d5f1404) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Threading.Tests.Perf_Timer Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [SynchronousContention - Duration of single invocation]() | 1.48 secs | 2.02 secs | 1.36 | 0.29 | True | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_Timer*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Threading.Tests.Perf_Timer.SynchronousContention ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.0187393852666666 > 1.5564687483725. IsChangePoint: Marked as a change because one of 6/12/2023 3:15:58 AM, 6/19/2023 11:51:49 PM falls between 6/10/2023 7:23:52 PM and 6/19/2023 11:51:49 PM. IsRegressionStdDev: Marked as regression because -14.06732025634985 (T) = (0 -2104758606.1809525) / Math.Sqrt((2732912057190156 / (17)) + (47891860143521784 / (28))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (28) - 2, .025) and -0.40669280758699733 = (1496246085.022215 - 2104758606.1809525) / 1496246085.022215 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `os-linux`, `arch-x64`, `area-GC-coreclr`, `untriaged`, `runtime-coreclr`
Milestone: -
cincuranet commented 1 year ago

More on Linux/arm64:

Maoni0 commented 1 year ago

it's interesting that this only regressed on linux but on windows?

@mrsharm do we have this particular test in our microbenchmark test suite?

mrsharm commented 1 year ago

it's interesting that this only regressed on linux but on windows?

@mrsharm do we have this particular test in our microbenchmark test suite?

We don't currently - seems like a new microbenchmark since we evaluated the set we care about and it does seem like a good candidate.

Also, I don't think it's necessarily true that we only regressed on linux; comparing the trends for this microbenchmark between the two we observe more noise across runs that have some regressions that weren't present before June 12th:

Windows: image Linux: image

We'll need to investigate further.

mangod9 commented 11 months ago

@markples since we have determined this wouldnt be fixed in 8, can we move this to 9 at the point?

mrsharm commented 1 month ago

We should probably move this issue and https://github.com/dotnet/runtime/issues/87626 to 10 since we know we won't be adding a fix, right @markples?