search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 10 Regressions on 5/19/2023 12:51:11 AM #18031

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Equals_Bytes_Chars - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Perf_Ascii.Equals_Bytes_Chars(Size%3a%20128).html>) 457.04 ns 721.28 ns 1.58 0.22 True
[EqualsIgnoreCase_ExactlyTheSame_Bytes_Chars - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Bytes_Chars(Size%3a%20128).html>) 469.83 ns 694.81 ns 1.48 0.12 True
[EqualsIgnoreCase_DifferentCase_Bytes_Chars - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Perf_Ascii.EqualsIgnoreCase_DifferentCase_Bytes_Chars(Size%3a%20128).html>) 698.11 ns 1.20 μs 1.72 0.10 True
[EqualsIgnoreCase_ExactlyTheSame_Chars - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Chars(Size%3a%206).html>) 92.52 ns 107.25 ns 1.16 0.20 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Perf_Ascii.Equals_Bytes_Chars(Size: 128) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 721.2803028297801 > 478.5697786483088. IsChangePoint: Marked as a change because one of 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -12.105737553043916 (T) = (0 -730.1258350461768) / Math.Sqrt((6407.585054509968 / (26)) + (1897.4330121124788 / (19))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (19) - 2, .025) and -0.44623736111217055 = (504.84509298301003 - 730.1258350461768) / 504.84509298301003 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Bytes_Chars(Size: 128) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 694.814801908125 > 493.3073403443475. IsChangePoint: Marked as a change because one of 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -10.049202697064144 (T) = (0 -710.5236718738362) / Math.Sqrt((5138.100220846952 / (25)) + (1517.8405196779413 / (19))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (19) - 2, .025) and -0.3139559529253439 = (540.7515147611699 - 710.5236718738362) / 540.7515147611699 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Text.Perf_Ascii.EqualsIgnoreCase_DifferentCase_Bytes_Chars(Size: 128) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1981392551424266 > 733.8513321732098. IsChangePoint: Marked as a change because one of 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -20.560308088869956 (T) = (0 -1215.9905630347318) / Math.Sqrt((8543.066825792688 / (26)) + (2137.951215705636 / (19))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (19) - 2, .025) and -0.5506650962943983 = (784.1735561989283 - 1215.9905630347318) / 784.1735561989283 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` ### JIT Disasms #### System.Text.Perf_Ascii.EqualsIgnoreCase_ExactlyTheSame_Chars(Size: 6) ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 107.24786936942068 > 96.05272028854307. IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -11.327494152690239 (T) = (0 -104.10775853354886) / Math.Sqrt((9.112244341540677 / (26)) + (8.594813973035436 / (19))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (19) - 2, .025) and -0.10802161395920788 = (93.95823801807319 - 104.10775853354886) / 93.95823801807319 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EqualsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Vector4.EqualsBenchmark.html>) 44.82 ns 161.40 ns 3.60 0.26 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Vector4.EqualsBenchmark ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 161.40208204462039 > 46.46762023369487. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/2/2023 11:15:04 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -31.849344187048434 (T) = (0 -162.07247255775735) / Math.Sqrt((301.564223238414 / (25)) + (18.359633191964274 / (19))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (19) - 2, .025) and -2.4402542038715387 = (47.110609551865906 - 162.07247255775735) / 47.110609551865906 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EqualsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Vector3.EqualsBenchmark.html>) 42.88 ns 181.11 ns 4.22 0.29 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Vector3.EqualsBenchmark ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 181.10667911763113 > 46.89404347651365. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/2/2023 11:15:04 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -20.125561452339355 (T) = (0 -180.769055365185) / Math.Sqrt((829.0624776111072 / (26)) + (146.75783541423843 / (19))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (19) - 2, .025) and -2.3411352886248733 = (54.104081322485136 - 180.769055365185) / 54.104081322485136 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Matrix3x2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EqualsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark.html>) 162.18 ns 737.22 ns 4.55 0.25 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix3x2*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 737.2244018009587 > 171.27305176105668. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/2/2023 11:15:04 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -35.86993391042634 (T) = (0 -743.9997219354364) / Math.Sqrt((5219.24528377472 / (24)) + (484.28843716258064 / (19))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (19) - 2, .025) and -3.023973964432667 = (184.8917832251262 - 743.9997219354364) / 184.8917832251262 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EqualsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Vector2.EqualsBenchmark.html>) 44.85 ns 170.50 ns 3.80 0.34 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Vector2.EqualsBenchmark ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 170.49661310332402 > 47.00880840616012. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/2/2023 11:15:04 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -29.64806403039111 (T) = (0 -178.1213551148657) / Math.Sqrt((318.36364305649545 / (26)) + (132.8887307799425 / (19))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (19) - 2, .025) and -2.704801604767176 = (48.078513809124615 - 178.1213551148657) / 48.078513809124615 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EqualsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Matrix4x4.EqualsBenchmark.html>) 297.71 ns 791.40 ns 2.66 0.23 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Matrix4x4.EqualsBenchmark ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 791.3990624268885 > 314.92148655740874. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/2/2023 11:15:04 AM, 3/16/2023 12:15:41 AM, 3/16/2023 10:20:48 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -33.2941396103443 (T) = (0 -809.9248685336283) / Math.Sqrt((4539.254343631835 / (25)) + (763.5782315174905 / (19))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (19) - 2, .025) and -1.5783664698468505 = (314.1232551716111 - 809.9248685336283) / 314.1232551716111 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline aef327f3b418bf5dd4f25083aff160c9cdf4b159
Compare 2f49fcff6df15a200ef01eea16b3ce7930f75c5c
Diff Diff
Configs CompilationMode:wasm, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Quaternion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EqualsBenchmark - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompilationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark.html>) 63.17 ns 162.03 ns 2.56 0.26 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark ```log ``` ### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 162.02969826731444 > 66.73839345022085. IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/2/2023 11:15:04 AM, 3/17/2023 10:06:19 AM, 5/18/2023 7:10:13 PM, 5/23/2023 2:36:40 AM falls between 5/14/2023 5:56:01 AM and 5/23/2023 2:36:40 AM. IsRegressionStdDev: Marked as regression because -35.22640229692167 (T) = (0 -166.37180313726515) / Math.Sqrt((3.553203917439853 / (26)) + (158.42157583000204 / (19))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (19) - 2, .025) and -1.6067524465823098 = (63.8233996309829 - 166.37180313726515) / 63.8233996309829 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

the range is https://github.com/dotnet/runtime/compare/f73356c1f4b9b292dc4fb02f1dd47e4a61e423f5...cb9894975ff60bdabe7e388a4a6868f578816b83

@kg looks like one of your changes. I see that it improved partially after, could you take a look?

kg commented 1 year ago

Yeah, I think the codegen is bad for certain uses of vectors which causes this. It improved some with optimizations I made later to 16-byte-sized copies but it's still kind of bad. A few of these are at least really blatant examples we can hopefully look at and fix.