search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 7 Regressions on 1/15/2023 10:25:39 PM #11832

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 191b98d3a63b7b2d8fd2c8cdf7c1f083160251d8
Compare 2908445dae75580421a3585a45ba3d570bd75cd3
Diff Diff

Regressions in System.Collections.CopyTo<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Array - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.CopyTo(String).Array(Size%3a%202048).html>) 10.74 μs 12.07 μs 1.12 0.46 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CopyTo<String>.Array(Size: 2048) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.072850654163446 > 10.4494441908901. IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 12/14/2022 11:14:18 PM, 1/15/2023 6:48:05 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -12.681376212285942 (T) = (0 -11983.61751735723) / Math.Sqrt((741905.2110877576 / (48)) + (15620.702014191123 / (9))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (9) - 2, .025) and -0.16110673380451423 = (10320.85782337286 - 11983.61751735723) / 10320.85782337286 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 191b98d3a63b7b2d8fd2c8cdf7c1f083160251d8
Compare 2908445dae75580421a3585a45ba3d570bd75cd3
Diff Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetValue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetValue.html>) 16.13 ns 26.76 ns 1.66 0.70 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetValue ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 26.760579983633118 > 17.191122567728367. IsChangePoint: Marked as a change because one of 1/15/2023 6:48:05 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -9.029944649446318 (T) = (0 -23.64007773746966) / Math.Sqrt((6.340326585144901 / (48)) + (3.6277265206921276 / (9))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (9) - 2, .025) and -0.38780185397169037 = (17.034188036148777 - 23.64007773746966) / 17.034188036148777 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 191b98d3a63b7b2d8fd2c8cdf7c1f083160251d8
Compare 2908445dae75580421a3585a45ba3d570bd75cd3
Diff Diff

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[SequenceCompareTo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).SequenceCompareTo(Size%3a%204).html>) 22.13 ns 26.31 ns 1.19 0.62 False
[Reverse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).Reverse(Size%3a%20512).html>) 165.41 ns 181.96 ns 1.10 0.01 False
[SequenceEqual - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).SequenceEqual(Size%3a%20512).html>) 293.54 ns 343.35 ns 1.17 0.00 False
[StartsWith - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).StartsWith(Size%3a%20512).html>) 149.94 ns 176.59 ns 1.18 0.03 False
[EndsWith - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Int32).EndsWith(Size%3a%20512).html>) 151.26 ns 180.99 ns 1.20 0.01 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Int32>.SequenceCompareTo(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 26.312439779835426 > 24.191300824994723. IsChangePoint: Marked as a change because one of 1/15/2023 6:48:05 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -5.870242255845162 (T) = (0 -27.522525662323424) / Math.Sqrt((5.308472408770693 / (48)) + (1.9486017086800658 / (9))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (9) - 2, .025) and -0.13893436468167636 = (24.16515517996137 - 27.522525662323424) / 24.16515517996137 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Int32>.Reverse(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 181.95694441033578 > 173.7995406284211. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -50.51740625270476 (T) = (0 -182.53042579164222) / Math.Sqrt((0.9138595753631271 / (32)) + (2.0009853139507703 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.10035742715164268 = (165.8828497792175 - 182.53042579164222) / 165.8828497792175 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Int32>.SequenceEqual(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 343.34756124679626 > 305.8587698276505. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -105.1614766004193 (T) = (0 -344.7834951205122) / Math.Sqrt((4.226453323000013 / (32)) + (3.2336142982785088 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.1847615478136906 = (291.0150956171402 - 344.7834951205122) / 291.0150956171402 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Int32>.StartsWith(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 176.59061864004147 > 159.84796660224285. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -53.198567627023515 (T) = (0 -176.57980354561616) / Math.Sqrt((6.0620377575141315 / (32)) + (0.4582313092911449 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.1591844814347065 = (152.33106237504646 - 176.57980354561616) / 152.33106237504646 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Int32>.EndsWith(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 180.9865922212758 > 158.42406336186522. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM. IsRegressionStdDev: Marked as regression because -66.81280140786993 (T) = (0 -177.5978575850546) / Math.Sqrt((1.899932918204738 / (32)) + (2.427253129269053 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.17482380285805468 = (151.16978150511005 - 177.5978575850546) / 151.16978150511005 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

This looks like effect of https://github.com/dotnet/runtime/pull/80668

cc @EgorBo

EgorBo commented 1 year ago

This looks like effect of dotnet/runtime#80668

cc @EgorBo

https://github.com/dotnet/runtime/issues/80762