search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 6 Regressions on 9/23/2022 6:26:22 PM #8719

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8e4a4cfdd6f7fd0b643dfd932baf5f8356d346e
Compare 7681692bdcb3ab4290194f2ead2bedd929af76d5
Diff Diff

Regressions in System.IO.Tests.Perf_FileStream

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Write_NoBuffering - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize%3a%201048576%2c%20userBufferSize%3a%2016384%2c%20options%3a%20Asynchronous).html>) 748.23 μs 818.64 μs 1.09 0.02 False
[CopyToFile - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.CopyToFile(fileSize%3a%201048576%2c%20options%3a%20None).html>) 685.09 μs 745.74 μs 1.09 0.02 False
[Write_NoBuffering - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize%3a%201048576%2c%20userBufferSize%3a%2016384%2c%20options%3a%20None).html>) 725.88 μs 814.39 μs 1.12 0.03 False
[Write_NoBuffering_PreallocationSize - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_FileStream.Write_NoBuffering_PreallocationSize(fileSize%3a%201048576%2c%20userBufferSize%3a%2016384%2c%20options%3a%20None).html>) 745.50 μs 841.46 μs 1.13 0.05 False

graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_FileStream*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: Asynchronous) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 818.6398000469925 > 783.0778451822915. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/23/2022 4:51:48 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -11.992374578999755 (T) = (0 -785474.1191491935) / Math.Sqrt((159711936.1287712 / (37)) + (398306381.7065908 / (19))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (19) - 2, .025) and -0.08314760904372463 = (725177.3558754958 - 785474.1191491935) / 725177.3558754958 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_FileStream.CopyToFile(fileSize: 1048576, options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 745.7387304421768 > 726.6667860869567. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/23/2022 4:51:48 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -14.088152338991076 (T) = (0 -742013.6346849703) / Math.Sqrt((125589412.32604782 / (37)) + (108447215.34856361 / (19))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (19) - 2, .025) and -0.06076167541580451 = (699510.2216472053 - 742013.6346849703) / 699510.2216472053 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 814.388155111336 > 755.1429594733391. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/23/2022 4:51:48 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -15.60267898282515 (T) = (0 -786635.9564208342) / Math.Sqrt((128475100.22218192 / (38)) + (241444878.5476279 / (20))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (20) - 2, .025) and -0.0845648034150471 = (725301.0183844221 - 786635.9564208342) / 725301.0183844221 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.IO.Tests.Perf_FileStream.Write_NoBuffering_PreallocationSize(fileSize: 1048576, userBufferSize: 16384, options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 841.4559289529914 > 778.4614201822917. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/23/2022 4:51:48 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -14.864189867853211 (T) = (0 -807089.2052248761) / Math.Sqrt((158004460.2784776 / (38)) + (271937548.5940494 / (20))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (20) - 2, .025) and -0.08413192507621792 = (744456.6353565643 - 807089.2052248761) / 744456.6353565643 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9b9b0c95c678065044d01049c56915e6b750b10a
Compare 7681692bdcb3ab4290194f2ead2bedd929af76d5
Diff Diff

Regressions in System.Collections.CtorDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Dictionary - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CtorDefaultSize(String).Dictionary.html>) 143.19 ns 163.89 ns 1.14 0.09 False
[HashSet - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CtorDefaultSize(String).HashSet.html>) 137.03 ns 155.60 ns 1.14 0.09 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorDefaultSize&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Collections.CtorDefaultSize<String>.Dictionary ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 163.88705576876768 > 150.2669493947986. IsChangePoint: Marked as a change because one of 9/23/2022 2:49:56 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -18.015197770906735 (T) = (0 -164.11011780015866) / Math.Sqrt((28.717995654645165 / (39)) + (13.582445911155869 / (19))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (19) - 2, .025) and -0.1523956532436347 = (142.40778966689072 - 164.11011780015866) / 142.40778966689072 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CtorDefaultSize<String>.HashSet ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 155.59618179127997 > 142.3709543594545. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/23/2022 6:26:22 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -30.893372705818127 (T) = (0 -155.067198635234) / Math.Sqrt((11.079409768534152 / (40)) + (2.4880185985248207 / (17))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (17) - 2, .025) and -0.1489304120602042 = (134.96657152383608 - 155.067198635234) / 134.96657152383608 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9b9b0c95c678065044d01049c56915e6b750b10a
Compare 7681692bdcb3ab4290194f2ead2bedd929af76d5
Diff Diff

Improvements in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EmptyStaticFunction5Arg - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.EmptyStaticFunction5Arg.html>) 33.15 ms 30.02 ms 0.91 0.16 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### PerfLabTests.LowLevelPerf.EmptyStaticFunction5Arg ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 30.022258341666664 < 31.31873312895833. IsChangePoint: Marked as a change because one of 7/9/2022 7:00:36 PM, 7/19/2022 10:33:10 AM, 9/23/2022 2:49:56 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsImprovementStdDev: Marked as improvement because 31.91473310563188 (T) = (0 -30189147.587293956) / Math.Sqrt((1543966024441.1865 / (299)) + (108178727102.2367 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.10051538010915181 = (33562716.82661721 - 30189147.587293956) / 33562716.82661721 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9b9b0c95c678065044d01049c56915e6b750b10a
Compare 7681692bdcb3ab4290194f2ead2bedd929af76d5
Diff Diff

Regressions in Microsoft.Extensions.DependencyInjection.GetService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[NoDI - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.DependencyInjection.GetService.NoDI.html>) 14.54 ns 15.97 ns 1.10 0.51 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.DependencyInjection.GetService*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Microsoft.Extensions.DependencyInjection.GetService.NoDI ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 15.97276467049781 > 15.291540964532874. IsChangePoint: Marked as a change because one of 7/15/2022 11:01:25 AM, 9/23/2022 2:49:56 PM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -6.16637328193458 (T) = (0 -16.202700639183576) / Math.Sqrt((1.4836182074964472 / (87)) + (0.866695359464029 / (40))) is less than -1.9791241094234997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (87) + (40) - 2, .025) and -0.08095029181106565 = (14.989311499270654 - 16.202700639183576) / 14.989311499270654 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9b9b0c95c678065044d01049c56915e6b750b10a
Compare 7681692bdcb3ab4290194f2ead2bedd929af76d5
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[CreateScaleFromScalarBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarBenchmark.html>) 20.27 ns 23.18 ns 1.14 0.57 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarBenchmark ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 23.17814357400609 > 22.277962515605523. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/23/2022 2:49:56 PM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -8.1761943613374 (T) = (0 -25.190667105818427) / Math.Sqrt((4.079098375113624 / (86)) + (6.34886362361887 / (40))) is less than -1.9792801166046092 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (86) + (40) - 2, .025) and -0.17284039420577527 = (21.478342006524304 - 25.190667105818427) / 21.478342006524304 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)