search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 3 Regressions on 9/29/2022 7:42:19 AM #8794

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 70b33f8f69128eeb1463691e24646987b54a0e3d
Compare 561b0974793ba91936f151eab07875938e853694
Diff Diff

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Subtract - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Decimal.Subtract.html>) 17.03 ns 19.90 ns 1.17 0.51 False
[Mod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Decimal.Mod.html>) 24.76 ns 27.31 ns 1.10 0.41 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Decimal*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_Decimal.Subtract ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 19.897852353412702 > 18.323029098551384. IsChangePoint: Marked as a change because one of 9/29/2022 2:21:18 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -2.5826014984678336 (T) = (0 -21.41138111859655) / Math.Sqrt((2.2327998824112445 / (120)) + (11.219930539583725 / (6))) is less than -1.9792801166046092 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (120) + (6) - 2, .025) and -0.19869713884557635 = (17.862210916109394 - 21.41138111859655) / 17.862210916109394 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Decimal.Mod ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 27.312972043377897 > 25.993044901269975. IsChangePoint: Marked as a change because one of 7/11/2022 1:18:03 PM, 7/20/2022 1:22:04 AM, 9/23/2022 4:51:48 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -8.117980545496657 (T) = (0 -28.086050437892855) / Math.Sqrt((2.7076862415631164 / (85)) + (1.7529431671334996 / (41))) is less than -1.9792801166046092 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (41) - 2, .025) and -0.08571808750275252 = (25.86864008362728 - 28.086050437892855) / 25.86864008362728 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 70b33f8f69128eeb1463691e24646987b54a0e3d
Compare 561b0974793ba91936f151eab07875938e853694
Diff Diff

Regressions in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Queue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.ContainsTrue(String).Queue(Size%3a%20512).html>) 2.71 ms 2.98 ms 1.10 0.36 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Collections.ContainsTrue<String>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 2.9784918162202385 > 2.734371417043269. IsChangePoint: Marked as a change because one of 7/15/2022 11:01:25 AM, 9/29/2022 2:21:18 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -5.919560761596668 (T) = (0 -2905659.5827819756) / Math.Sqrt((20205000262.34722 / (118)) + (18094730413.219845 / (6))) is less than -1.979599878486223 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (118) + (6) - 2, .025) and -0.12995672773988518 = (2571478.63404806 - 2905659.5827819756) / 2571478.63404806 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 70b33f8f69128eeb1463691e24646987b54a0e3d
Compare 561b0974793ba91936f151eab07875938e853694
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Narrow_long - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_VectorConvert.Narrow_long.html>) 241.95 μs 257.26 μs 1.06 0.10 False
[Narrow_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong.html>) 237.82 μs 261.60 μs 1.10 0.08 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Numerics.Tests.Perf_VectorConvert.Narrow_long ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 257.2584162763466 > 254.85760022096898. IsChangePoint: Marked as a change because one of 7/9/2022 7:00:36 PM, 7/12/2022 12:16:14 PM, 7/19/2022 2:55:40 PM, 9/29/2022 2:21:18 AM, 10/3/2022 1:11:02 PM falls between 9/24/2022 8:58:25 PM and 10/3/2022 1:11:02 PM. IsRegressionStdDev: Marked as regression because -17.353462315630352 (T) = (0 -258905.77017201806) / Math.Sqrt((33579006.408477694 / (41)) + (3575458.4511074563 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.07112711923484162 = (241713.39285757893 - 258905.77017201806) / 241713.39285757893 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 261.59599350961537 > 250.48853792067314. IsChangePoint: Marked as a change because one of 7/9/2022 7:00:36 PM, 7/12/2022 7:47:24 PM, 7/19/2022 10:33:10 AM, 9/29/2022 2:21:18 AM, 10/3/2022 1:11:02 PM falls between 9/24/2022 8:58:25 PM and 10/3/2022 1:11:02 PM. IsRegressionStdDev: Marked as regression because -11.766464846874431 (T) = (0 -260656.428932207) / Math.Sqrt((40884075.464214996 / (41)) + (42656503.66465759 / (23))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (23) - 2, .025) and -0.08252288868060943 = (240786.0671194656 - 260656.428932207) / 240786.0671194656 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 70b33f8f69128eeb1463691e24646987b54a0e3d
Compare 561b0974793ba91936f151eab07875938e853694
Diff Diff

Improvements in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Int16.ToString(value%3a%2032767).html>) 73.86 ns 66.78 ns 0.90 0.37 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int16*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_Int16.ToString(value: 32767) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 66.77587716175577 < 67.04534632675052. IsChangePoint: Marked as a change because one of 9/29/2022 2:21:18 AM, 10/3/2022 1:11:02 PM falls between 9/24/2022 8:58:25 PM and 10/3/2022 1:11:02 PM. IsImprovementStdDev: Marked as improvement because 6.638618582872484 (T) = (0 -65.62935121618527) / Math.Sqrt((14.172742862891194 / (299)) + (6.710795097127251 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.05563291767302247 = (69.49559386850989 - 65.62935121618527) / 69.49559386850989 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)