dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.21k stars 4.72k forks source link

[Perf] Windows/arm64: 1 Regression on 1/13/2023 8:11:05 PM #80861

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture arm64
OS Windows 10.0.25094
Baseline b53dfdf794d0355f2b615290f6763ae34ba88e8a
Compare 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Diff Diff

Regressions in System.Threading.Tests.Perf_Thread

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetCurrentProcessorId - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.25094/System.Threading.Tests.Perf_Thread.GetCurrentProcessorId.html>) 1.93 ns 5.59 ns 2.90 0.89 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_Thread*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Threading.Tests.Perf_Thread.GetCurrentProcessorId ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.590015446048096 > 2.899760743523029. IsChangePoint: Marked as a change because one of 10/5/2022 3:48:29 AM, 10/11/2022 5:13:36 PM, 1/13/2023 3:30:06 PM, 1/19/2023 1:09:52 AM falls between 1/10/2023 12:53:48 PM and 1/19/2023 1:09:52 AM. IsRegressionStdDev: Marked as regression because -4.081587394244101 (T) = (0 -8.168585037970475) / Math.Sqrt((2.040771031822769 / (23)) + (26.661786250261294 / (20))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (20) - 2, .025) and -1.4740550483169508 = (3.3016989834269843 - 8.168585037970475) / 3.3016989834269843 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
ghost commented 1 year ago

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Architecture | arm64 -- | -- OS | Windows 10.0.25094 Baseline | [b53dfdf794d0355f2b615290f6763ae34ba88e8a](https://github.com/dotnet/runtime/commit/b53dfdf794d0355f2b615290f6763ae34ba88e8a) Compare | [99ba09612b31e90e1e7feb5511ee541f9f3ef02b](https://github.com/dotnet/runtime/commit/99ba09612b31e90e1e7feb5511ee541f9f3ef02b) Diff | [Diff](https://github.com/dotnet/runtime/compare/b53dfdf794d0355f2b615290f6763ae34ba88e8a...99ba09612b31e90e1e7feb5511ee541f9f3ef02b) ### Regressions in System.Threading.Tests.Perf_Thread Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [GetCurrentProcessorId - Duration of single invocation]() | 1.93 ns | 5.59 ns | 2.90 | 0.89 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_Thread*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Threading.Tests.Perf_Thread.GetCurrentProcessorId ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.590015446048096 > 2.899760743523029. IsChangePoint: Marked as a change because one of 10/5/2022 3:48:29 AM, 10/11/2022 5:13:36 PM, 1/13/2023 3:30:06 PM, 1/19/2023 1:09:52 AM falls between 1/10/2023 12:53:48 PM and 1/19/2023 1:09:52 AM. IsRegressionStdDev: Marked as regression because -4.081587394244101 (T) = (0 -8.168585037970475) / Math.Sqrt((2.040771031822769 / (23)) + (26.661786250261294 / (20))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (20) - 2, .025) and -1.4740550483169508 = (3.3016989834269843 - 8.168585037970475) / 3.3016989834269843 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: -
Labels: `area-System.Threading`
Milestone: -
cincuranet commented 1 year ago

Looks PGO related.

cc @AndyAyersMS

mangod9 commented 1 year ago

this is mis-labeled into System.Threading. Is there is lable for PGO?