search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Changes at 6/27/2022 5:19:23 PM #6480

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[WriteBasicUtf8 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted%3a%20True%2c%20SkipValidation%3a%20False%2c%20DataSize%3a%20100000).html>) 4.10 ms 4.44 ms 1.08 0.29 False
[WriteBasicUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted%3a%20False%2c%20SkipValidation%3a%20False%2c%20DataSize%3a%20100000).html>) 3.00 ms 3.30 ms 1.10 0.14 False
[WriteBasicUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted%3a%20True%2c%20SkipValidation%3a%20False%2c%20DataSize%3a%20100000).html>) 4.03 ms 4.90 ms 1.22 0.13 False
[WriteBasicUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted%3a%20True%2c%20SkipValidation%3a%20True%2c%20DataSize%3a%20100000).html>) 3.57 ms 4.26 ms 1.19 0.22 False
[WriteBasicUtf16 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted%3a%20False%2c%20SkipValidation%3a%20True%2c%20DataSize%3a%20100000).html>) 2.69 ms 2.98 ms 1.11 0.16 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.444458431168831 > 4.311377064942185. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/14/2022 4:58:13 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -16.83379344648698 (T) = (0 -4463305.100109552) / Math.Sqrt((9142080237.480173 / (32)) + (827201494.6939982 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.07645483707550935 = (4146300.3800840997 - 4463305.100109552) / 4146300.3800840997 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.3006775093984952 > 3.1589532336930657. IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/16/2022 8:46:33 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -19.042326597587493 (T) = (0 -3263586.6601551957) / Math.Sqrt((3596978411.389027 / (32)) + (871565799.7410324 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08621198296409414 = (3004557.776327788 - 3263586.6601551957) / 3004557.776327788 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.897048922222223 > 4.247385092811829. IsChangePoint: Marked as a change because one of 3/17/2022 11:30:59 AM, 4/14/2022 9:05:43 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -9.517592618394199 (T) = (0 -4504264.109989512) / Math.Sqrt((7036560955.811689 / (32)) + (21098221125.94473 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.10372562352431872 = (4080963.61449587 - 4504264.109989512) / 4080963.61449587 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.258219942690059 > 3.7959758508970585. IsChangePoint: Marked as a change because one of 3/14/2022 7:02:48 PM, 4/14/2022 9:05:43 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -13.527137559910468 (T) = (0 -4344886.498613778) / Math.Sqrt((22752946990.1049 / (32)) + (20337299786.97004 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1802286339097474 = (3681393.9043492423 - 4344886.498613778) / 3681393.9043492423 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.981186115662651 > 2.838501074550744. IsChangePoint: Marked as a change because one of 3/17/2022 11:30:59 AM, 4/16/2022 2:55:52 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -13.43886572168155 (T) = (0 -3037361.6314837974) / Math.Sqrt((5921492961.505787 / (32)) + (4220622328.004373 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.11421621236995425 = (2726007.3922486594 - 3037361.6314837974) / 2726007.3922486594 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FormatterInt64 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value%3a%209223372036854775807).html>) 47.75 ns 58.05 ns 1.22 0.40 False
[FormatterInt32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value%3a%2012345).html>) 13.48 ns 23.88 ns 1.77 0.76 False
[FormatterInt32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value%3a%202147483647).html>) 20.80 ns 27.50 ns 1.32 0.72 False
[FormatterInt32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value%3a%20-2147483648).html>) 18.13 ns 23.85 ns 1.32 0.69 False
[FormatterInt64 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value%3a%20-9223372036854775808).html>) 44.67 ns 50.71 ns 1.14 0.48 False
[FormatterInt64 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value%3a%2012345).html>) 16.18 ns 22.72 ns 1.40 0.76 False

graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 58.049817872416355 > 49.32882021594207. IsChangePoint: Marked as a change because one of 4/15/2022 3:54:24 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -7.452668879009599 (T) = (0 -53.986856534069894) / Math.Sqrt((6.588462114017324 / (32)) + (6.497378260181185 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.13551064776552363 = (47.544121792522255 - 53.986856534069894) / 47.544121792522255 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.87913629143294 > 13.948645017615885. IsChangePoint: Marked as a change because one of 4/14/2022 9:05:43 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -12.419496327778193 (T) = (0 -22.256574943366417) / Math.Sqrt((3.5330661656749265 / (32)) + (4.42466545747915 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.6293374420188959 = (13.659892892284189 - 22.256574943366417) / 13.659892892284189 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.501393730681837 > 23.304146599893283. IsChangePoint: Marked as a change because one of 4/14/2022 9:05:43 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -9.081961084525158 (T) = (0 -30.03402578391474) / Math.Sqrt((6.864684459288596 / (32)) + (6.933184085420094 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.3682847837321641 = (21.95012773729257 - 30.03402578391474) / 21.95012773729257 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 23.853929626161055 > 15.981084372597541. IsChangePoint: Marked as a change because one of 4/14/2022 4:58:13 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -13.3297358933596 (T) = (0 -24.888331841447478) / Math.Sqrt((3.6207930546753615 / (32)) + (3.923815476524818 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5511547266636977 = (16.045034975317108 - 24.888331841447478) / 16.045034975317108 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 50.7111355236821 > 46.29760386082962. IsChangePoint: Marked as a change because one of 4/14/2022 9:05:43 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -10.741645099401214 (T) = (0 -51.879611449806625) / Math.Sqrt((6.5820841116265 / (32)) + (4.838650830898245 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.19270026991908473 = (43.49761021964575 - 51.879611449806625) / 43.49761021964575 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 22.722251919388587 > 14.126255740541126. IsChangePoint: Marked as a change because one of 4/14/2022 9:05:43 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -9.663538969944737 (T) = (0 -21.542573484412348) / Math.Sqrt((4.406705104294888 / (32)) + (4.366090843656918 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.46559750339779227 = (14.698833366233748 - 21.542573484412348) / 14.698833366233748 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Memory.Slice<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[MemoryStartLengthSpan - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Memory.Slice(Byte).MemoryStartLengthSpan.html>) 9.24 ns 10.58 ns 1.14 0.42 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Slice&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Slice<Byte>.MemoryStartLengthSpan ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.576191238946981 > 6.634014670785989. IsChangePoint: Marked as a change because one of 3/17/2022 11:30:59 AM, 5/20/2022 10:18:01 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -8.971207503998814 (T) = (0 -11.684089223684104) / Math.Sqrt((1.5327994833127516 / (32)) + (2.7893696885360533 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.6850363645735776 = (6.934027935142473 - 11.684089223684104) / 6.934027935142473 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Collections.Concurrent.Count<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Queue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu 18.04/System.Collections.Concurrent.Count(String).Queue(Size%3a%20512).html>) 39.66 ns 56.60 ns 1.43 0.64 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.Count&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Concurrent.Count<String>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 56.60361586920631 > 53.58270992560154. IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM. IsRegressionStdDev: Marked as regression because -11.72357655439461 (T) = (0 -56.59333294525117) / Math.Sqrt((28.77074481077868 / (32)) + (3.2671018915573513 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.28899331364477726 = (43.90506323514355 - 56.59333294525117) / 43.90506323514355 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
dakersnar commented 2 years ago

@AndyAyersMS Do you think all of this is PGO or is some of it worth looking closer at?

dakersnar commented 2 years ago

Almost every test here follows a similar pattern that looks like this image

AndyAyersMS commented 2 years ago

Could be PGO but could also be https://github.com/dotnet/runtime/pull/71161. Eg the latter has ARM64 diffs like:

         -56 (-2.33 % of base) : 2952.dasm - System.Text.Json.Tests.Perf_Basic:WriteBasicUtf16():this