dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 6/27/2022 5:19:23 PM #6468

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FirstSpanTenSegments - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSpanTenSegments.html>) 5.69 ns 7.02 ns 1.23 0.05 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.ReadOnlySequenceTests<Byte>.FirstSpanTenSegments ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.0162097975051525 > 5.9703434236123805. IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -470.6217250326128 (T) = (0 -7.019055383345071) / Math.Sqrt((0.0001323035815207426 / (32)) + (4.211005575665659E-05 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.23335085999914257 = (5.691045112134556 - 7.019055383345071) / 5.691045112134556 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FormatterInt32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value%3a%204).html>) 1.57 ns 3.14 ns 2.00 0.66 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.142487415555736 > 1.562328386948982. IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -28.899622499417553 (T) = (0 -3.2134171468903774) / Math.Sqrt((0.03596237484130557 / (32)) + (0.030101091177771034 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -1.266403453603855 = (1.4178486808165847 - 3.2134171468903774) / 1.4178486808165847 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryFormat3 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Tests.Perf_Version.TryFormat3.html>) 16.29 ns 19.75 ns 1.21 0.11 False
[TryFormat2 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Tests.Perf_Version.TryFormat2.html>) 9.90 ns 12.38 ns 1.25 0.21 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Version*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Version.TryFormat3 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.74600593073405 > 17.121778285691605. IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -21.529141968687583 (T) = (0 -19.890795769229324) / Math.Sqrt((0.11465599031931947 / (32)) + (0.2664909405641053 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.22025226089788627 = (16.300560471482573 - 19.890795769229324) / 16.300560471482573 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Version.TryFormat2 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.38180292406811 > 10.36166581621029. IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -14.01570500273899 (T) = (0 -12.606417213770976) / Math.Sqrt((0.12044688001676762 / (32)) + (0.3897237266834753 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.28222372593970524 = (9.831683004096725 - 12.606417213770976) / 9.831683004096725 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Collections.Concurrent.Count<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Queue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Collections.Concurrent.Count(String).Queue(Size%3a%20512).html>) 7.36 ns 11.99 ns 1.63 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.Count&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Concurrent.Count<String>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 11.993115261873474 > 7.759746036804716. IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -80.48124249780878 (T) = (0 -12.13688403153791) / Math.Sqrt((0.07103548235659639 / (32)) + (0.013319893712199419 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.6350634995323768 = (7.422882374298626 - 12.13688403153791) / 7.422882374298626 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetBytes - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Text.Tests.Perf_Encoding.GetBytes(size%3a%2016%2c%20encName%3a%20%22ascii%22).html>) 24.04 ns 27.25 ns 1.13 0.12 False
[GetByteCount - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Text.Tests.Perf_Encoding.GetByteCount(size%3a%2016%2c%20encName%3a%20%22utf-8%22).html>) 13.96 ns 15.37 ns 1.10 0.11 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Tests.Perf_Encoding*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "ascii") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.25313664733826 > 25.438431131139602. IsChangePoint: Marked as a change because one of 5/24/2022 10:45:34 PM, 6/8/2022 9:45:10 AM, 6/22/2022 5:33:38 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -12.335323184128606 (T) = (0 -27.46295901909355) / Math.Sqrt((0.7741685899684854 / (32)) + (0.20628059167842577 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.10263462808893356 = (24.90667200130641 - 27.46295901909355) / 24.90667200130641 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "utf-8") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.373752744393332 > 14.652991377979806. IsChangePoint: Marked as a change because one of 3/9/2022 4:16:21 PM, 3/22/2022 12:26:23 PM, 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -10.060190424243945 (T) = (0 -15.529871218566639) / Math.Sqrt((0.32382339043585984 / (16)) + (0.009066132060632857 / (26))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (26) - 2, .025) and -0.10247308964057111 = (14.086394819514094 - 15.529871218566639) / 14.086394819514094 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FirstSpanTenSegments - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows 10.0.19041/System.Buffers.Tests.ReadOnlySequenceTests(Char).FirstSpanTenSegments.html>) 6.36 ns 8.23 ns 1.29 0.04 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.ReadOnlySequenceTests<Char>.FirstSpanTenSegments ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.225797854046279 > 6.67080668973954. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/17/2022 8:30:53 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -257.7427770581941 (T) = (0 -8.187831161386097) / Math.Sqrt((0.000243667010327193 / (32)) + (0.0004772839132123639 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.2900100175947695 = (6.347106650111408 - 8.187831161386097) / 6.347106650111408 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
dakersnar commented 2 years ago

@EgorBo Take a look at this one

AndyAyersMS commented 2 years ago

Think this one might be PGO.