search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/arm64: 6 Regressions on 5/2/2023 7:50:39 PM #17649

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 3e8f17a65a068fca3d19fa5cd43a7e1cd414a5ae
Compare c5303ae33513e1a2eddceb2986639efd7505115e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchmark.GetChildKeysTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[AddChainedConfigurationEmpty - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty.html>) 24.05 ms 37.73 ms 1.57 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchmark.GetChildKeysTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 37.73028214285714 > 25.047024951923078. IsChangePoint: Marked as a change because one of 1/27/2023 3:43:27 AM, 4/17/2023 8:43:19 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM. IsRegressionStdDev: Marked as regression because -343.85492830011344 (T) = (0 -37825917.09761905) / Math.Sqrt((12203069797.271664 / (9)) + (7304233373.274289 / (25))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (25) - 2, .025) and -0.5848857589318825 = (23866652.144764956 - 37825917.09761905) / 23866652.144764956 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 3e8f17a65a068fca3d19fa5cd43a7e1cd414a5ae
Compare c5303ae33513e1a2eddceb2986639efd7505115e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SingleSerial - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleSerial(RentalSize%3a%204096%2c%20ManipulateArray%3a%20True%2c%20Async%3a%20True%2c%20UseSharedPool%3a%20False).html>) 1.79 μs 2.11 μs 1.18 0.03 False
[SingleParallel - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleParallel(RentalSize%3a%204096%2c%20ManipulateArray%3a%20True%2c%20Async%3a%20True%2c%20UseSharedPool%3a%20True).html>) 3.06 μs 3.26 μs 1.07 0.01 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.SingleSerial(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: False) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.114772 > 1.9254863137500002. IsChangePoint: Marked as a change because one of 1/18/2023 8:37:25 PM, 3/17/2023 1:30:52 PM, 4/8/2023 3:10:58 PM, 4/25/2023 6:46:56 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM. IsRegressionStdDev: Marked as regression because -24.932104652678596 (T) = (0 -2114.2107572837417) / Math.Sqrt((1107.6491696520452 / (9)) + (28.101407814754555 / (26))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (26) - 2, .025) and -0.1512754422697382 = (1836.4074136034533 - 2114.2107572837417) / 1836.4074136034533 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.SingleParallel(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: True) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.2601994999999997 > 3.2213747825000003. IsChangePoint: Marked as a change because one of 1/18/2023 8:37:25 PM, 3/17/2023 1:30:52 PM, 4/8/2023 3:10:58 PM, 4/25/2023 6:46:56 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM. IsRegressionStdDev: Marked as regression because -15.131024404782073 (T) = (0 -3247.2342654722092) / Math.Sqrt((1720.1206766057019 / (12)) + (110.80634551770763 / (23))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (23) - 2, .025) and -0.06012848305707492 = (3063.0572778388273 - 3247.2342654722092) / 3063.0572778388273 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 3e8f17a65a068fca3d19fa5cd43a7e1cd414a5ae
Compare c5303ae33513e1a2eddceb2986639efd7505115e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[HeapSort - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).HeapSort(Size%3a%201000).html>) 75.36 μs 85.04 μs 1.13 0.02 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 1000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 85.04142027712723 > 79.0943048610787. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM. IsRegressionStdDev: Marked as regression because -5.8727237565098465 (T) = (0 -84540.4847868098) / Math.Sqrt((13576513.926884264 / (9)) + (8079898.311758055 / (25))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (25) - 2, .025) and -0.10377223703612512 = (76592.32761083018 - 84540.4847868098) / 76592.32761083018 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 3e8f17a65a068fca3d19fa5cd43a7e1cd414a5ae
Compare c5303ae33513e1a2eddceb2986639efd7505115e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryParse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Tests.Perf_UInt16.TryParse(value%3a%20%220%22).html>) 10.32 ns 11.62 ns 1.13 0.32 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_UInt16.TryParse(value: "0") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.620709559486743 > 10.8282976267314. IsChangePoint: Marked as a change because one of 4/6/2023 1:55:21 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM. IsRegressionStdDev: Marked as regression because -21.28710778982039 (T) = (0 -11.60326055349633) / Math.Sqrt((0.0142637120140727 / (9)) + (0.053962653925045546 / (26))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (26) - 2, .025) and -0.12485116639769647 = (10.315374069136132 - 11.60326055349633) / 10.315374069136132 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 3e8f17a65a068fca3d19fa5cd43a7e1cd414a5ae
Compare c5303ae33513e1a2eddceb2986639efd7505115e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString_MultipleSegments - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length%3a%20100).html>) 33.97 ns 38.02 ns 1.12 0.12 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 38.02468774732282 > 35.85759213496294. IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 4/10/2023 1:33:42 PM, 4/25/2023 6:46:56 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM. IsRegressionStdDev: Marked as regression because -20.325596267079135 (T) = (0 -37.70610107564704) / Math.Sqrt((0.2131772599401625 / (9)) + (0.19904497253249062 / (26))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (26) - 2, .025) and -0.1055002360242954 = (34.10772774798248 - 37.70610107564704) / 34.10772774798248 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
EgorBo commented 1 year ago

PGO