search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.17k stars 4.72k forks source link

Regressions in System.Collections.Tests.Perf_BitArray #85472

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[BitArraySetAll - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size%3a%20512).html>) 6.34 ns 20.21 ns 3.19 0.08 True
[BitArrayOr - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Tests.Perf_BitArray.BitArrayOr(Size%3a%20512).html>) 10.23 ns 13.27 ns 1.30 0.17 False
[BitArrayNot - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Tests.Perf_BitArray.BitArrayNot(Size%3a%20512).html>) 7.68 ns 12.34 ns 1.61 0.09 True
[BitArrayXor - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Tests.Perf_BitArray.BitArrayXor(Size%3a%20512).html>) 11.10 ns 13.27 ns 1.20 0.16 False
[BitArrayAnd - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size%3a%20512).html>) 10.21 ns 13.27 ns 1.30 0.14 False

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 20.212927410149806 > 6.74401509131666. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -22.782032921087676 (T) = (0 -19.388217620025838) / Math.Sqrt((0.764127380030929 / (12)) + (9.154316463812194 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -1.9018141528755539 = (6.681412591779207 - 19.388217620025838) / 6.681412591779207 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.270547236239794 > 10.721935688096798. IsChangePoint: Marked as a change because one of 3/3/2023 11:43:56 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -21.55662134780645 (T) = (0 -13.090932598882455) / Math.Sqrt((0.08404562154238614 / (12)) + (0.3654561295071419 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2721811755099 = (10.29014801577732 - 13.090932598882455) / 10.29014801577732 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.34367551357869 > 8.45729261356312. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -25.173315248612905 (T) = (0 -12.225031627889667) / Math.Sqrt((0.13336744721792546 / (12)) + (0.5985314200161592 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.515541865454447 = (8.066442707093344 - 12.225031627889667) / 8.066442707093344 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.27152853578668 > 11.195543888775145. IsChangePoint: Marked as a change because one of 3/3/2023 11:43:56 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -18.26129989179763 (T) = (0 -13.129635689054156) / Math.Sqrt((0.14109544984139194 / (12)) + (0.37006676689544954 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.25812483042935563 = (10.435876767946352 - 13.129635689054156) / 10.435876767946352 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.274841288890011 > 10.724252428267. IsChangePoint: Marked as a change because one of 3/3/2023 11:43:56 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -26.332988000240494 (T) = (0 -13.155049980522081) / Math.Sqrt((0.06340106453117673 / (12)) + (0.24686904533672896 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2802018694796814 = (10.275762201369657 - 13.155049980522081) / 10.275762201369657 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToStringWithFormat - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Double.ToStringWithFormat(value%3a%20-1.7976931348623157E%2b308%2c%20format%3a%20%22G17%22).html>) 173.10 ns 189.76 ns 1.10 0.04 False
[ToStringWithFormat - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Double.ToStringWithFormat(value%3a%201.7976931348623157E%2b308%2c%20format%3a%20%22G%22).html>) 187.03 ns 199.07 ns 1.06 0.05 False
[ToStringWithFormat - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Double.ToStringWithFormat(value%3a%20-1.7976931348623157E%2b308%2c%20format%3a%20%22G%22).html>) 189.85 ns 201.99 ns 1.06 0.04 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G17") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 189.76186173085713 > 182.12087727615054. IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/16/2023 8:14:39 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -16.280367616050867 (T) = (0 -193.74048682864122) / Math.Sqrt((2.921473629852635 / (12)) + (50.95241369335574 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.1197871718227939 = (173.01545481474815 - 193.74048682864122) / 173.01545481474815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 199.0659946216465 > 195.76746533251998. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -16.593279560293134 (T) = (0 -202.28571375792214) / Math.Sqrt((7.2385776634357315 / (11)) + (18.670439273809272 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.0970313887972808 = (184.39373369224742 - 202.28571375792214) / 184.39373369224742 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 201.9929318081847 > 199.571868700985. IsChangePoint: Marked as a change because one of 2/26/2023 1:18:12 AM, 3/7/2023 7:00:18 AM, 3/10/2023 8:23:08 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -16.05519271839466 (T) = (0 -204.15310026562997) / Math.Sqrt((9.116719032127556 / (12)) + (13.46376130056293 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.09094311799718856 = (187.13450490473335 - 204.15310026562997) / 187.13450490473335 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString_Flags - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Enum.ToString_Flags(value%3a%2036).html>) 21.12 ns 23.11 ns 1.09 0.09 False
[ToString_Flags - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Enum.ToString_Flags(value%3a%2032).html>) 20.57 ns 23.82 ns 1.16 0.09 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Enum.ToString_Flags(value: 36) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.109282559966346 > 22.207627074377847. IsChangePoint: Marked as a change because one of 3/5/2023 11:54:18 PM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -24.416003491257246 (T) = (0 -23.52948663947281) / Math.Sqrt((0.030306352843840333 / (12)) + (0.2097208620111538 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.10366664268316057 = (21.31937827011741 - 23.52948663947281) / 21.31937827011741 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Enum.ToString_Flags(value: 32) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.819586554127373 > 22.06371874051919. IsChangePoint: Marked as a change because one of 2/17/2023 9:53:52 PM, 2/23/2023 8:54:44 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -14.446217882630442 (T) = (0 -23.327030207622716) / Math.Sqrt((0.10704885295620135 / (12)) + (0.689660119866922 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.11459126981740116 = (20.928775273328927 - 23.327030207622716) / 20.928775273328927 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[LeadingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong.html>) 426.99 ns 453.99 ns 1.06 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 453.9888571400803 > 437.4870198443056. IsChangePoint: Marked as a change because one of 3/6/2023 11:36:59 AM, 3/20/2023 8:31:51 AM, 4/11/2023 3:40:00 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -6.899473543431359 (T) = (0 -449.63921276123074) / Math.Sqrt((127.73613522789564 / (12)) + (24.48687560196992 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.054402916989556856 = (426.43965178416147 - 449.63921276123074) / 426.43965178416147 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[WriteDeepUtf16 - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted%3a%20False%2c%20SkipValidation%3a%20True).html>) 4.69 ms 5.52 ms 1.18 0.08 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.5245486956521725 > 4.963877521952104. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -24.485068782269277 (T) = (0 -5330890.328456869) / Math.Sqrt((5260312330.949635 / (11)) + (3955702275.6601996 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.12498930332550309 = (4738614.236329708 - 5330890.328456869) / 4738614.236329708 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EndsWith - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Memory.Span(Int32).EndsWith(Size%3a%20512).html>) 17.70 ns 19.94 ns 1.13 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Int32>.EndsWith(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.940756126078455 > 18.415689949805408. IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/3/2023 11:43:56 AM, 3/6/2023 11:14:52 AM, 3/14/2023 8:42:09 PM, 3/29/2023 12:02:44 PM, 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -35.91000189497899 (T) = (0 -20.492056053295332) / Math.Sqrt((0.038686993493706835 / (11)) + (0.09548884472250842 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.1585348877302128 = (17.687905880368536 - 20.492056053295332) / 17.687905880368536 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Subtract - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Numerics.Tests.Perf_BigInteger.Subtract(arguments%3a%2016%2c16%20bits).html>) 9.94 ns 12.09 ns 1.22 0.06 False
[ToStringX - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Numerics.Tests.Perf_BigInteger.ToStringX(numberString%3a%20123).html>) 57.93 ns 62.57 ns 1.08 0.04 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 16,16 bits) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.093060336348236 > 10.521474691525725. IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -77.16242824139755 (T) = (0 -12.085444119274811) / Math.Sqrt((0.006091211191040216 / (12)) + (0.007867817031197065 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2067816437762143 = (10.0146071839952 - 12.085444119274811) / 10.0146071839952 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Numerics.Tests.Perf_BigInteger.ToStringX(numberString: 123) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 62.56798798274013 > 60.847873618535985. IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 2/10/2023 8:30:00 AM, 2/24/2023 8:06:06 PM, 3/22/2023 7:51:01 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -14.40322671948516 (T) = (0 -62.92725860755048) / Math.Sqrt((0.9228104554270762 / (12)) + (0.5065039595240335 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.07398793721799111 = (58.59214654733865 - 62.92725860755048) / 58.59214654733865 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetBytes - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Text.Tests.Perf_Encoding.GetBytes(size%3a%2016%2c%20encName%3a%20%22utf-8%22).html>) 17.05 ns 18.85 ns 1.11 0.03 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "utf-8") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 18.846580134218538 > 17.9414329037105. IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 8:42:09 PM, 3/25/2023 12:18:49 AM, 4/5/2023 6:15:03 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -11.462439215091996 (T) = (0 -19.55628530717003) / Math.Sqrt((0.37676268568474464 / (12)) + (0.08885202522279244 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.12090212313203894 = (17.44691610764873 - 19.55628530717003) / 17.44691610764873 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToStringWithFormat - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Single.ToStringWithFormat(value%3a%203.4028235E%2b38%2c%20format%3a%20%22G17%22).html>) 164.68 ns 181.63 ns 1.10 0.04 False
[ToStringWithFormat - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Tests.Perf_Single.ToStringWithFormat(value%3a%20-3.4028235E%2b38%2c%20format%3a%20%22G17%22).html>) 168.46 ns 190.61 ns 1.13 0.06 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Single*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Single.ToStringWithFormat(value: 3.4028235E+38, format: "G17") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 181.63333167990325 > 173.21449436062247. IsChangePoint: Marked as a change because one of 2/17/2023 9:53:52 PM, 2/23/2023 8:54:44 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -10.615428136554076 (T) = (0 -192.92588894924864) / Math.Sqrt((26.942965029812935 / (12)) + (80.16547356032582 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.13168168811846295 = (170.47716771843128 - 192.92588894924864) / 170.47716771843128 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Single.ToStringWithFormat(value: -3.4028235E+38, format: "G17") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 190.60926047735387 > 176.61243956619884. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -11.918662808225005 (T) = (0 -195.35381811169196) / Math.Sqrt((22.693571317953644 / (11)) + (59.186693849490936 / (35))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (35) - 2, .025) and -0.13405838436949685 = (172.26081196895603 - 195.35381811169196) / 172.26081196895603 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ICollection - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsTrue(String).ICollection(Size%3a%20512).html>) 418.24 ÎĽs 450.05 ÎĽs 1.08 0.08 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsTrue<String>.ICollection(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 450.04744047619045 > 439.81782319078945. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -6.915511891108824 (T) = (0 -456612.5545593939) / Math.Sqrt((177884375.83685806 / (12)) + (196994040.0356963 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.07294011480620924 = (425571.33269443066 - 456612.5545593939) / 425571.33269443066 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ScalarValueTypeObj - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/PerfLabTests.CastingPerf.ScalarValueTypeObj.html>) 278.52 ÎĽs 306.35 ÎĽs 1.10 0.03 False
[ObjrefValueTypeObj - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/PerfLabTests.CastingPerf.ObjrefValueTypeObj.html>) 278.41 ÎĽs 306.31 ÎĽs 1.10 0.04 False
[IntObj - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/PerfLabTests.CastingPerf.IntObj.html>) 278.44 ÎĽs 306.46 ÎĽs 1.10 0.04 False
[FooObjIsFoo - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/PerfLabTests.CastingPerf.FooObjIsFoo.html>) 278.60 ÎĽs 306.49 ÎĽs 1.10 0.04 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.CastingPerf.ScalarValueTypeObj ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 306.3475773001508 > 292.4505637770433. IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -656.046552854986 (T) = (0 -306386.778021153) / Math.Sqrt((18672.635718482783 / (12)) + (9129.638899069914 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.10000787621105743 = (278531.4402261306 - 306386.778021153) / 278531.4402261306 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### PerfLabTests.CastingPerf.ObjrefValueTypeObj ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 306.31208858543414 > 292.3966451322115. IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -838.0905066332441 (T) = (0 -306354.00740025734) / Math.Sqrt((11261.843632552698 / (12)) + (6152.321257190443 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.10002410963038778 = (278497.53902502503 - 306354.00740025734) / 278497.53902502503 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### PerfLabTests.CastingPerf.IntObj ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 306.46116946778716 > 292.441201171875. IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -807.9901910210531 (T) = (0 -306357.76509096357) / Math.Sqrt((11614.127680945023 / (12)) + (8004.306362643904 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.09981733987841093 = (278553.31424837565 - 306357.76509096357) / 278553.31424837565 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### PerfLabTests.CastingPerf.FooObjIsFoo ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 306.4854901960784 > 292.4477490234375. IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -1257.1749801102367 (T) = (0 -306346.8586710498) / Math.Sqrt((3167.0400773902365 / (12)) + (8388.488591019073 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.09998851443242886 = (278500.0521838343 - 306346.8586710498) / 278500.0521838343 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Hypot - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.MathBenchmarks.Single.Hypot.html>) 15.32 ÎĽs 16.72 ÎĽs 1.09 0.02 False
[Exp - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.MathBenchmarks.Single.Exp.html>) 19.50 ÎĽs 20.89 ÎĽs 1.07 0.03 False

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.MathBenchmarks.Single.Hypot ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.719442513368985 > 16.089785437773. IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/22/2023 7:51:01 PM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -783.3359869112949 (T) = (0 -16718.61600943784) / Math.Sqrt((28.311748540684718 / (12)) + (28.667436583432703 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.09078861578740795 = (15327.090664004745 - 16718.61600943784) / 15327.090664004745 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.MathBenchmarks.Single.Exp ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 20.893140040106957 > 20.48001608696249. IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -603.6473618458264 (T) = (0 -20895.809779150302) / Math.Sqrt((54.82220569687902 / (12)) + (27.985913790487007 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.07142342787260947 = (19502.8494202712 - 20895.809779150302) / 19502.8494202712 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FrozenDictionaryOptimized - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.TryGetValueFalse(Int32%2c%20Int32).FrozenDictionaryOptimized(Size%3a%20512).html>) 2.02 ÎĽs 2.30 ÎĽs 1.14 0.03 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.TryGetValueFalse<Int32, Int32>.FrozenDictionaryOptimized(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.3003796309944966 > 2.121923287750782. IsChangePoint: Marked as a change because one of 4/5/2023 11:58:06 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -20.180275088602635 (T) = (0 -2299.50903272552) / Math.Sqrt((2088.2312870667547 / (12)) + (0.7936887967574187 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.13093476004065316 = (2033.2817718352387 - 2299.50903272552) / 2033.2817718352387 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Span - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.ContainsFalse(String).Span(Size%3a%20512).html>) 712.71 ÎĽs 814.97 ÎĽs 1.14 0.19 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsFalse<String>.Span(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 814.9746428571428 > 748.3568026200772. IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -6.787256941935116 (T) = (0 -818622.7801391255) / Math.Sqrt((1582264920.3870263 / (12)) + (1448672606.3195305 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.12160754453806688 = (729865.6148717996 - 818622.7801391255) / 729865.6148717996 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EmptyTakeSelectToArray - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray.html>) 17.54 ns 19.10 ns 1.09 0.12 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.097756083601173 > 17.840828734250543. IsChangePoint: Marked as a change because one of 2/13/2023 6:54:14 PM, 2/20/2023 9:09:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -18.39626057261227 (T) = (0 -18.91344323686354) / Math.Sqrt((0.10282578512782935 / (12)) + (0.05821642135092264 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.10859189897705936 = (17.060780666280987 - 18.91344323686354) / 17.060780666280987 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IntObj - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/PerfLabTests.CastingPerf2.CastingPerf.IntObj.html>) 111.38 ÎĽs 167.04 ÎĽs 1.50 0.02 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.CastingPerf2.CastingPerf.IntObj ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 167.0387172667758 > 116.9559088243317. IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -4011.392829358727 (T) = (0 -167102.0992753773) / Math.Sqrt((1532.2055413495834 / (12)) + (2403.8234130497426 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.49969152043087367 = (111424.31426655495 - 167102.0992753773) / 111424.31426655495 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FrozenDictionary - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Collections.IterateForEach(String).FrozenDictionary(Size%3a%20512).html>) 2.44 ÎĽs 2.59 ÎĽs 1.06 0.09 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.IterateForEach<String>.FrozenDictionary(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.5893050945901095 > 2.565480535160109. IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -21.879440030515823 (T) = (0 -2602.0185011971757) / Math.Sqrt((1.0390532795249057 / (12)) + (1941.5098888042512 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.0649186091188095 = (2443.3965928628795 - 2602.0185011971757) / 2443.3965928628795 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetFileNameWithoutExtension - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.IO.Tests.Perf_Path.GetFileNameWithoutExtension.html>) 27.57 ns 30.38 ns 1.10 0.07 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_Path*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_Path.GetFileNameWithoutExtension ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 30.384713429173438 > 28.857825995642738. IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/14/2023 11:21:41 AM, 3/17/2023 9:25:32 PM, 3/20/2023 8:31:51 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -26.520003044153448 (T) = (0 -30.48910070212556) / Math.Sqrt((0.03620090670624483 / (11)) + (0.3421020278843808 / (36))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (36) - 2, .025) and -0.10912093296671935 = (27.489428605925 - 30.48910070212556) / 27.489428605925 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryParseSingle - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value%3a%2012345).html>) 31.07 ns 32.98 ns 1.06 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: 12345) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 32.97781688421849 > 32.62557112859975. IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -32.028029721462815 (T) = (0 -33.40991687844801) / Math.Sqrt((0.034736520337854834 / (12)) + (0.07747592802288111 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.07262648704158192 = (31.147764186390845 - 33.40991687844801) / 31.147764186390845 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[WriteAsciiCharArray - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars%3a%202000000).html>) 81.54 ÎĽs 88.34 ÎĽs 1.08 0.00 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 88.34201586787565 > 84.78663289304123. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -33.90762173483388 (T) = (0 -87747.52615838259) / Math.Sqrt((131260.25173041897 / (12)) + (1032703.3432310184 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.08244403724199545 = (81064.26118985162 - 87747.52615838259) / 81064.26118985162 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in SciMark2.kernel

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[benchMonteCarlo - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/SciMark2.kernel.benchMonteCarlo.html>) 732.58 ms 776.09 ms 1.06 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'
### Payloads [Baseline]() [Compare]() ### Histogram #### SciMark2.kernel.benchMonteCarlo ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 776.0866733333334 > 769.085247. IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -244.50389085779807 (T) = (0 -776852159.0733589) / Math.Sqrt((314764066031.21893 / (12)) + (234489354084.88895 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.060219624096299135 = (732727579.661172 - 776852159.0733589) / 732727579.661172 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IterateTryGetArray - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows 10.0.19042/amd/System.Buffers.Tests.ReadOnlySequenceTests(Char).IterateTryGetArray.html>) 6.89 ns 7.97 ns 1.16 0.01 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateTryGetArray ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.968825114812262 > 7.235449130888102. IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -954.1864951779512 (T) = (0 -7.971466174293845) / Math.Sqrt((1.0790463871684801E-05 / (12)) + (1.395053752640102E-05 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.1563724083317204 = (6.893511222560342 - 7.971466174293845) / 6.893511222560342 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
kunalspathak commented 1 year ago

@cincuranet - can you check the commit range between 4/15 and 4/17 of when it regressed? @AndyAyersMS - most likely from https://github.com/dotnet/runtime/pull/84875.

cincuranet commented 1 year ago

@kunalspathak The regression first occurred between https://github.com/dotnet/runtime/compare/9c02b25f08c8f1aaf7811446325ac9f873efa451...e92289a03ef47cd3ea2ada2cfe1c0c3abd04f196 or possibly https://github.com/dotnet/runtime/compare/2d833f43d4158c39ff62e57bf0f8d58b93935ae4...e92289a03ef47cd3ea2ada2cfe1c0c3abd04f196 (taking System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) and System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)).

kunalspathak commented 1 year ago

Thanks @cincuranet for finding out. @AndyAyersMS - could this be from https://github.com/dotnet/runtime/pull/84817?

ghost commented 1 year ago

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Collections.Tests.Perf_BitArray Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [BitArraySetAll - Duration of single invocation]() | 6.34 ns | 20.21 ns | 3.19 | 0.08 | True | | | [BitArrayOr - Duration of single invocation]() | 10.23 ns | 13.27 ns | 1.30 | 0.17 | False | | | [BitArrayNot - Duration of single invocation]() | 7.68 ns | 12.34 ns | 1.61 | 0.09 | True | | | [BitArrayXor - Duration of single invocation]() | 11.10 ns | 13.27 ns | 1.20 | 0.16 | False | | | [BitArrayAnd - Duration of single invocation]() | 10.21 ns | 13.27 ns | 1.30 | 0.14 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 20.212927410149806 > 6.74401509131666. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -22.782032921087676 (T) = (0 -19.388217620025838) / Math.Sqrt((0.764127380030929 / (12)) + (9.154316463812194 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -1.9018141528755539 = (6.681412591779207 - 19.388217620025838) / 6.681412591779207 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.270547236239794 > 10.721935688096798. IsChangePoint: Marked as a change because one of 3/3/2023 11:43:56 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -21.55662134780645 (T) = (0 -13.090932598882455) / Math.Sqrt((0.08404562154238614 / (12)) + (0.3654561295071419 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2721811755099 = (10.29014801577732 - 13.090932598882455) / 10.29014801577732 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.34367551357869 > 8.45729261356312. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -25.173315248612905 (T) = (0 -12.225031627889667) / Math.Sqrt((0.13336744721792546 / (12)) + (0.5985314200161592 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.515541865454447 = (8.066442707093344 - 12.225031627889667) / 8.066442707093344 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.27152853578668 > 11.195543888775145. IsChangePoint: Marked as a change because one of 3/3/2023 11:43:56 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -18.26129989179763 (T) = (0 -13.129635689054156) / Math.Sqrt((0.14109544984139194 / (12)) + (0.37006676689544954 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.25812483042935563 = (10.435876767946352 - 13.129635689054156) / 10.435876767946352 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.274841288890011 > 10.724252428267. IsChangePoint: Marked as a change because one of 3/3/2023 11:43:56 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -26.332988000240494 (T) = (0 -13.155049980522081) / Math.Sqrt((0.06340106453117673 / (12)) + (0.24686904533672896 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2802018694796814 = (10.275762201369657 - 13.155049980522081) / 10.275762201369657 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Tests.Perf_Double Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ToStringWithFormat - Duration of single invocation]() | 173.10 ns | 189.76 ns | 1.10 | 0.04 | False | | | [ToStringWithFormat - Duration of single invocation]() | 187.03 ns | 199.07 ns | 1.06 | 0.05 | False | | | [ToStringWithFormat - Duration of single invocation]() | 189.85 ns | 201.99 ns | 1.06 | 0.04 | False | | | ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G17") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 189.76186173085713 > 182.12087727615054. IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/16/2023 8:14:39 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -16.280367616050867 (T) = (0 -193.74048682864122) / Math.Sqrt((2.921473629852635 / (12)) + (50.95241369335574 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.1197871718227939 = (173.01545481474815 - 193.74048682864122) / 173.01545481474815 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 199.0659946216465 > 195.76746533251998. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -16.593279560293134 (T) = (0 -202.28571375792214) / Math.Sqrt((7.2385776634357315 / (11)) + (18.670439273809272 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.0970313887972808 = (184.39373369224742 - 202.28571375792214) / 184.39373369224742 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 201.9929318081847 > 199.571868700985. IsChangePoint: Marked as a change because one of 2/26/2023 1:18:12 AM, 3/7/2023 7:00:18 AM, 3/10/2023 8:23:08 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -16.05519271839466 (T) = (0 -204.15310026562997) / Math.Sqrt((9.116719032127556 / (12)) + (13.46376130056293 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.09094311799718856 = (187.13450490473335 - 204.15310026562997) / 187.13450490473335 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Tests.Perf_Enum Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ToString_Flags - Duration of single invocation]() | 21.12 ns | 23.11 ns | 1.09 | 0.09 | False | | | [ToString_Flags - Duration of single invocation]() | 20.57 ns | 23.82 ns | 1.16 | 0.09 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Enum.ToString_Flags(value: 36) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.109282559966346 > 22.207627074377847. IsChangePoint: Marked as a change because one of 3/5/2023 11:54:18 PM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -24.416003491257246 (T) = (0 -23.52948663947281) / Math.Sqrt((0.030306352843840333 / (12)) + (0.2097208620111538 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.10366664268316057 = (21.31937827011741 - 23.52948663947281) / 21.31937827011741 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Enum.ToString_Flags(value: 32) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.819586554127373 > 22.06371874051919. IsChangePoint: Marked as a change because one of 2/17/2023 9:53:52 PM, 2/23/2023 8:54:44 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -14.446217882630442 (T) = (0 -23.327030207622716) / Math.Sqrt((0.10704885295620135 / (12)) + (0.689660119866922 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.11459126981740116 = (20.928775273328927 - 23.327030207622716) / 20.928775273328927 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Numerics.Tests.Perf_BitOperations Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [LeadingZeroCount_ulong - Duration of single invocation]() | 426.99 ns | 453.99 ns | 1.06 | 0.02 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 453.9888571400803 > 437.4870198443056. IsChangePoint: Marked as a change because one of 3/6/2023 11:36:59 AM, 3/20/2023 8:31:51 AM, 4/11/2023 3:40:00 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -6.899473543431359 (T) = (0 -449.63921276123074) / Math.Sqrt((127.73613522789564 / (12)) + (24.48687560196992 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.054402916989556856 = (426.43965178416147 - 449.63921276123074) / 426.43965178416147 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Text.Json.Tests.Perf_Deep Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [WriteDeepUtf16 - Duration of single invocation]() | 4.69 ms | 5.52 ms | 1.18 | 0.08 | True | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.5245486956521725 > 4.963877521952104. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -24.485068782269277 (T) = (0 -5330890.328456869) / Math.Sqrt((5260312330.949635 / (11)) + (3955702275.6601996 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.12498930332550309 = (4738614.236329708 - 5330890.328456869) / 4738614.236329708 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Memory.Span<Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [EndsWith - Duration of single invocation]() | 17.70 ns | 19.94 ns | 1.13 | 0.02 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Int32>.EndsWith(Size: 512) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.940756126078455 > 18.415689949805408. IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/3/2023 11:43:56 AM, 3/6/2023 11:14:52 AM, 3/14/2023 8:42:09 PM, 3/29/2023 12:02:44 PM, 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -35.91000189497899 (T) = (0 -20.492056053295332) / Math.Sqrt((0.038686993493706835 / (11)) + (0.09548884472250842 / (37))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (37) - 2, .025) and -0.1585348877302128 = (17.687905880368536 - 20.492056053295332) / 17.687905880368536 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Numerics.Tests.Perf_BigInteger Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Subtract - Duration of single invocation]() | 9.94 ns | 12.09 ns | 1.22 | 0.06 | False | | | [ToStringX - Duration of single invocation]() | 57.93 ns | 62.57 ns | 1.08 | 0.04 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 16,16 bits) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.093060336348236 > 10.521474691525725. IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -77.16242824139755 (T) = (0 -12.085444119274811) / Math.Sqrt((0.006091211191040216 / (12)) + (0.007867817031197065 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2067816437762143 = (10.0146071839952 - 12.085444119274811) / 10.0146071839952 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Numerics.Tests.Perf_BigInteger.ToStringX(numberString: 123) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 62.56798798274013 > 60.847873618535985. IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 2/10/2023 8:30:00 AM, 2/24/2023 8:06:06 PM, 3/22/2023 7:51:01 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -14.40322671948516 (T) = (0 -62.92725860755048) / Math.Sqrt((0.9228104554270762 / (12)) + (0.5065039595240335 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.07398793721799111 = (58.59214654733865 - 62.92725860755048) / 58.59214654733865 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Text.Tests.Perf_Encoding Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [GetBytes - Duration of single invocation]() | 17.05 ns | 18.85 ns | 1.11 | 0.03 | False | | | ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "utf-8") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 18.846580134218538 > 17.9414329037105. IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 8:42:09 PM, 3/25/2023 12:18:49 AM, 4/5/2023 6:15:03 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -11.462439215091996 (T) = (0 -19.55628530717003) / Math.Sqrt((0.37676268568474464 / (12)) + (0.08885202522279244 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.12090212313203894 = (17.44691610764873 - 19.55628530717003) / 17.44691610764873 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
--- ### Run Information Name | Value -- | -- Architecture | x64 OS | Windows 10.0.19042 Queue | OwlWindows Baseline | [7526a4c6a0a095670f1135d983a349b4e867f2e2](https://github.com/dotnet/runtime/commit/7526a4c6a0a095670f1135d983a349b4e867f2e2) Compare | [c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8](https://github.com/dotnet/runtime/commit/c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Diff | [Diff](https://github.com/dotnet/runtime/compare/7526a4c6a0a095670f1135d983a349b4e867f2e2...c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8) Configs | CompilationMode:tiered, RunKind:micro ### Regressions in System.Tests.Perf_Single Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ToStringWithFormat - Duration of single invocation]() | 164.68 ns | 181.63 ns | 1.10 | 0.04 | False | | | [ToStringWithFormat - Duration of single invocation]() | 168.46 ns | 190.61 ns | 1.13 | 0.06 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md ### Payloads [Baseline]() [Compare]() ```cmd git clone https://github.com/dotnet/performance.git py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Single*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Single.ToStringWithFormat(value: 3.4028235E+38, format: "G17") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 181.63333167990325 > 173.21449436062247. IsChangePoint: Marked as a change because one of 2/17/2023 9:53:52 PM, 2/23/2023 8:54:44 PM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -10.615428136554076 (T) = (0 -192.92588894924864) / Math.Sqrt((26.942965029812935 / (12)) + (80.16547356032582 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.13168168811846295 = (170.47716771843128 - 192.92588894924864) / 170.47716771843128 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms #### System.Tests.Perf_Single.ToStringWithFormat(value: -3.4028235E+38, format: "G17") ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 190.60926047735387 > 176.61243956619884. IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM. IsRegressionStdDev: Marked as regression because -11.918662808225005 (T) = (0 -195.35381811169196) / Math.Sqrt((22.693571317953644 / (11)) + (59.186693849490936 / (35))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (35) - 2, .025) and -0.13405838436949685 = (172.26081196895603 - 195.35381811169196) / 172.26081196895603 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: AndyAyersMS
Labels: `os-windows`, `arch-x64`, `area-CodeGen-coreclr`, `untriaged`, `runtime-coreclr`, `needs-area-label`
Milestone: -
AndyAyersMS commented 1 year ago

Note regression in System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) is amd64 (orange) only, not seeng this on intel (blue)

image

Similarly for System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

image
AndyAyersMS commented 1 year ago

Seems like this has also fixed itself (somewhat) at least the worst cases:

newplot - 2023-05-12T153319 162

newplot - 2023-05-12T153441 631

Will have to dig through some the others to see if there's anything that is not back to "normal" here.

newplot - 2023-05-15T114108 044

newplot - 2023-05-15T114506 618 newplot - 2023-05-15T114522 315

AndyAyersMS commented 1 year ago

Here's the full set, sorted from worst to least worst.

Note we are just evaluating the regression between 4/12 and 4/17 here.

Ratio Test Notes
3.19 System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512) Fixed itself
1.61 System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) Fixed itself
1.50 PerfLabTests.CastingPerf2.CastingPerf.IntObj Bistable
1.30 System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512) Fixed itself
1.30 System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512) Fixed itself
1.22 System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 16,16 bits) Still regressed Analysis
1.20 System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512) Fixed itself
1.18 System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True) Fixed itself
1.16 System.Tests.Perf_Enum.ToString_Flags(value: 32) Still regressed
1.16 System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateTryGetArray Fixed itself
1.14 System.Collections.TryGetValueFalse<Int32, Int32>.FrozenDictionaryOptimized(Size: 512) Bistable
1.14 System.Collections.ContainsFalse<String>.Span(Size: 512) Still regressed
1.13 System.Memory.Span<Int32>.EndsWith(Size: 512) Multi-stable
1.13 System.Tests.Perf_Single.ToStringWithFormat(value: -3.4028235E+38, format: "G17") Fixed itself
1.11 System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "utf-8") Still regressed
1.10 PerfLabTests.CastingPerf.ScalarValueTypeObj bistable
1.10 PerfLabTests.CastingPerf.ObjrefValueTypeObj bistable
1.10 PerfLabTests.CastingPerf.IntObj bistable
1.10 PerfLabTests.CastingPerf.FooObjIsFoo bistable
1.10 System.Tests.Perf_Single.ToStringWithFormat(value: 3.4028235E+38, format: "G17") Fixed itself
1.10 System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G17") Fixed itself
1.10 System.IO.Tests.Perf_Path.GetFileNameWithoutExtension Still regressed
1.09 System.Tests.Perf_Enum.ToString_Flags(value: 36) Still regressed
1.09 System.MathBenchmarks.Single.Hypot bistable
1.09 System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray Fixed itself
1.08 System.Numerics.Tests.Perf_BigInteger.ToStringX(numberString: 123) Fixed itself
1.08 System.Collections.ContainsTrue<String>.ICollection(Size: 512) Still regressed
1.08 System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000) Fixed itself
1.07 System.MathBenchmarks.Single.Exp bistable
1.06 System.Collections.IterateForEach<String>.FrozenDictionary(Size: 512) [Fixed itself] (https://github.com/dotnet/runtime/issues/85472#issuecomment-1548836769)
1.06 System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: 12345) Still regressed
1.06 SciMark2.kernel.benchMonteCarlo bistable
1.06 System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G") Fixed itself
1.06 System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G") Fixed itself
1.06 System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong noisy
AndyAyersMS commented 1 year ago

8 benchmarks that are still regressed as of 5/15

newplot - 2023-05-15T114840 280 newplot - 2023-05-15T115500 389 newplot - 2023-05-15T182840 176 newplot - 2023-05-15T163429 057 newplot - 2023-05-15T164244 428 newplot - 2023-05-15T181710 262 newplot - 2023-05-15T183332 859 newplot - 2023-05-15T183850 922

AndyAyersMS commented 1 year ago

More that are no longer regressed

newplot - 2023-05-15T115257 196

newplot - 2023-05-15T163108 289

newplot - 2023-05-15T164115 596

newplot - 2023-05-15T181425 014

newplot - 2023-05-15T181555 206

AndyAyersMS commented 1 year ago

More that are no longer regressed newplot - 2023-05-15T183102 286 newplot - 2023-05-15T183223 243 newplot - 2023-05-15T183447 321 newplot - 2023-05-15T183750 518 newplot - 2023-05-15T184144 207 newplot - 2023-05-15T184246 059

AndyAyersMS commented 1 year ago

System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 16,16 bits)

The regression on 4/14 is amd64 only, about 2ns slowdown.

Since then things have gotten even worse, the regression on 5/2 looks like it may be from https://github.com/dotnet/runtime/pull/85620 (see https://github.com/dotnet/perf-autofiling-issues/issues/17605) which was (hopefully) fixed by https://github.com/dotnet/runtime/pull/86246.

So we should check back in a day or two and see where this test has ended up.

image 
00.70%   2.6E+05     ?        Unknown
43.63%   1.624E+07   Tier-1   [System.Runtime.Numerics]BigInteger.Subtract(value class System.ReadOnlySpan`1<unsigned int32>,int32,value class System.ReadOnlySpan`1<unsigned int32>,int32)
28.10%   1.046E+07   Tier-1   [MicroBenchmarks]Perf_BigInteger.Subtract(class BigIntegers)
13.41%   4.99E+06    native   coreclr.dll (mostly checked write barrier)
08.92%   3.32E+06    Tier-1   [System.Runtime.Numerics]BigInteger..ctor(int64)
04.67%   1.74E+06    Tier-1   [fbbfaa49-80f2-4fb3-b341-975678a5e8cd]Runnable_0.WorkloadActionUnroll(int64)
00.32%   1.2E+05     native   clrjit.dll
00.11%   4E+04       native   ntoskrnl.exe
00.05%   2E+04       native   ntdll.dll

This seems to be largely fixed, indeed improved by PGO, but has recently back slid some.

newplot - 2023-06-08T185605 444

AndyAyersMS commented 1 year ago

System.Tests.Perf_Enum.ToString_Flags(value: 32)

The regression on 4/14 is amd64 only, about 4ns slowdown.

image
AndyAyersMS commented 1 year ago

System.Collections.ContainsFalse<String>.Span(Size: 512)

image
AndyAyersMS commented 1 year ago

Collated and updated. First few are bimodal, remainder are small.

Notes Recent Score Orig Score x64-win-amd Benchmark
1.50 1.50 1.50
1.50		 PerfLabTests.CastingPerf2.CastingPerf.IntObj
1.48 3.19 1.48
3.19		 System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512)
1.48 1.61 1.48
1.61		 System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512)
1.25 1.13 1.25
1.13		 System.Memory.Span(Int32).EndsWith(Size: 512)
1.25 1.22 1.25
1.22		 System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 16,16 bits)
1.10 1.10 1.10
1.10		 PerfLabTests.CastingPerf.FooObjIsFoo
1.10 1.10 1.10
1.10		 PerfLabTests.CastingPerf.IntObj
1.10 1.10 1.10
1.10		 PerfLabTests.CastingPerf.ObjrefValueTypeObj
1.10 1.10 1.10
1.10		 PerfLabTests.CastingPerf.ScalarValueTypeObj
1.07 1.14 1.07
1.14		 System.Collections.ContainsFalse(String).Span(Size: 512)
1.07 1.07 1.07
1.07		 System.MathBenchmarks.Single.Exp
1.06 1.06 1.06
1.06		 System.Collections.IterateForEach(String).FrozenDictionary(Size: 512)
1.04 1.30 1.04
1.30		 System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)
1.03 1.06 1.03
1.06		 System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong
1.01 1.06 1.01
1.06		 System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: 12345)
0.99 1.08 0.99
1.08		 System.Collections.ContainsTrue(String).ICollection(Size: 512)
0.97 1.08 0.97
1.08		 System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000)
0.94 1.30 0.94
1.30		 System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)
0.94 1.08 0.94
1.08		 System.Numerics.Tests.Perf_BigInteger.ToStringX(numberString: 123)
0.93 1.16 0.93
1.16		 System.Tests.Perf_Enum.ToString_Flags(value: 32)
0.93 1.09 0.93
1.09		 System.Tests.Perf_Enum.ToString_Flags(value: 36)
0.93 1.11 0.93
1.11		 System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "utf-8")
0.92 1.16 0.92
1.16		 System.Buffers.Tests.ReadOnlySequenceTests(Char).IterateTryGetArray
0.88 1.06 0.88
1.06		 SciMark2.kernel.benchMonteCarlo
0.87 1.06 0.87
1.06		 System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G")
0.87 1.20 0.87
1.20		 System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)
0.86 1.06 0.86
1.06		 System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G")
0.86 1.18 0.86
1.18		 System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)
0.83 1.10 0.83
1.10		 System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G17")
0.82 1.09 0.82
1.09		 System.MathBenchmarks.Single.Hypot
0.80 1.10 0.80
1.10		 System.Tests.Perf_Single.ToStringWithFormat(value: 3.4028235E+38, format: "G17")
0.78 1.09 0.78
1.09		 System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray
0.76 1.10 0.76
1.10		 System.IO.Tests.Perf_Path.GetFileNameWithoutExtension
0.74 1.13 0.74
1.13		 System.Tests.Perf_Single.ToStringWithFormat(value: -3.4028235E+38, format: "G17")
0.66 1.14 0.66
1.14		 System.Collections.TryGetValueFalse(Int32, Int32).FrozenDictionaryOptimized(Size: 512)