search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 22 Regressions on 1/19/2023 7:53:02 PM #12123

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IterateForEachArray - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Tests.ReadOnlySequenceTests(Char).IterateForEachArray.html>) 122.34 ns 168.58 ns 1.38 0.48 False
[IterateGetPositionTenSegments - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Tests.ReadOnlySequenceTests(Char).IterateGetPositionTenSegments.html>) 567.79 ns 649.16 ns 1.14 0.30 False
[SliceTenSegments - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Tests.ReadOnlySequenceTests(Char).SliceTenSegments.html>) 116.74 ns 123.93 ns 1.06 0.34 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateForEachArray ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 168.57730147382318 > 129.4312299220948. IsChangePoint: Marked as a change because one of 11/11/2022 3:15:52 PM, 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -7.991065657844798 (T) = (0 -150.12832464926575) / Math.Sqrt((39.48952697299523 / (32)) + (125.02324060651331 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.16277782883613565 = (129.11178810446907 - 150.12832464926575) / 129.11178810446907 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateGetPositionTenSegments ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 649.1550815292176 > 590.0567236408356. IsChangePoint: Marked as a change because one of 1/18/2023 3:58:17 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -8.399963335329797 (T) = (0 -611.4948611668126) / Math.Sqrt((221.78149977044868 / (31)) + (367.3784059836182 / (23))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (23) - 2, .025) and -0.07073393605606147 = (571.0987954852642 - 611.4948611668126) / 571.0987954852642 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Tests.ReadOnlySequenceTests<Char>.SliceTenSegments ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 123.93399282784121 > 123.78641176051869. IsChangePoint: Marked as a change because one of 1/17/2023 2:19:41 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -5.511648651675112 (T) = (0 -131.6193788119163) / Math.Sqrt((34.47185217413856 / (27)) + (68.24448449170157 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.08894142571645364 = (120.86910801957892 - 131.6193788119163) / 120.86910801957892 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[EmptyStaticFunction - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.EmptyStaticFunction.html>) 1.86 ms 2.08 ms 1.12 0.20 False
[GenericClassGenericInstanceMethod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod.html>) 288.07 μs 399.86 μs 1.39 0.06 False
[IntegerFormatting - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.IntegerFormatting.html>) 6.43 ms 16.00 ms 2.49 0.22 False
[GenericClassGenericStaticMethod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod.html>) 525.26 μs 654.43 μs 1.25 0.13 False
[GenericGenericMethod - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.LowLevelPerf.GenericGenericMethod.html>) 282.59 μs 407.45 μs 1.44 0.11 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.LowLevelPerf.EmptyStaticFunction ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.0777625026041666 > 1.9564735608333337. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -8.33904228229637 (T) = (0 -2092664.665958521) / Math.Sqrt((13224642035.058285 / (32)) + (4610432699.730079 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.11043203818290226 = (1884549.9715434476 - 2092664.665958521) / 1884549.9715434476 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 399.85781274038453 > 302.60327825. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -32.49655378639152 (T) = (0 -385545.59940948687) / Math.Sqrt((3722473.27127428 / (32)) + (193594183.73892084 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.33632609820108267 = (288511.61399039906 - 385545.59940948687) / 288511.61399039906 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.LowLevelPerf.IntegerFormatting ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 15.999866671428572 > 6.750741233177083. IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -85.51900747711608 (T) = (0 -15665665.741158683) / Math.Sqrt((88809789056.23221 / (14)) + (190736111824.69205 / (40))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (40) - 2, .025) and -1.3554463402301093 = (6650826.840584391 - 15665665.741158683) / 6650826.840584391 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 654.4288209635417 > 551.8575060757212. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -12.904246876630882 (T) = (0 -678746.1346516752) / Math.Sqrt((14108803.420922305 / (32)) + (3050603211.164123 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.28904395828650026 = (526550.0298018685 - 678746.1346516752) / 526550.0298018685 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.LowLevelPerf.GenericGenericMethod ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 407.4453983516483 > 296.680255409727. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -27.281880825488344 (T) = (0 -421779.54325366556) / Math.Sqrt((26778.137048615015 / (32)) + (572677012.4046113 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.49258026122844734 = (282584.1626141604 - 421779.54325366556) / 282584.1626141604 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Reverse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).Reverse(Size%3a%2033).html>) 24.07 ns 29.50 ns 1.23 0.33 False
[Reverse - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).Reverse(Size%3a%204).html>) 13.92 ns 16.34 ns 1.17 0.49 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Byte>.Reverse(Size: 33) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 29.496475509301526 > 25.27195062677713. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -6.369546646014871 (T) = (0 -27.27771003379253) / Math.Sqrt((2.5862794141959067 / (32)) + (2.5748570077647295 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.11590645135749313 = (24.44444155745257 - 27.27771003379253) / 24.44444155745257 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.Reverse(Size: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.335283971111732 > 14.595210687065808. IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/11/2023 4:46:06 PM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -7.246150159413201 (T) = (0 -16.257344985231487) / Math.Sqrt((0.6651465984984736 / (32)) + (1.7153993588341885 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.16288530149559166 = (13.98017926989261 - 16.257344985231487) / 13.98017926989261 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Collections.CtorDefaultSizeNonGeneric

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Hashtable - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.CtorDefaultSizeNonGeneric.Hashtable.html>) 33.20 ns 35.76 ns 1.08 0.55 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorDefaultSizeNonGeneric*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorDefaultSizeNonGeneric.Hashtable ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 35.75823674828893 > 34.60016184970855. IsChangePoint: Marked as a change because one of 12/5/2022 8:20:45 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -7.092898541732629 (T) = (0 -37.7738977748661) / Math.Sqrt((2.704561306230093 / (32)) + (5.5736301002327835 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1225169092093533 = (33.65107239361963 - 37.7738977748661) / 33.65107239361963 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Max - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Linq.Tests.Perf_Enumerable.Max(input%3a%20IEnumerable).html>) 2.20 μs 2.55 μs 1.16 0.24 False
[Min - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Linq.Tests.Perf_Enumerable.Min(input%3a%20IEnumerable).html>) 2.16 μs 2.53 μs 1.17 0.22 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.5542139839806897 > 2.2965649845619986. IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -15.545053296030265 (T) = (0 -2599.3393038626728) / Math.Sqrt((5964.80182199081 / (32)) + (8670.259667415621 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1683460289006842 = (2224.802618029552 - 2599.3393038626728) / 2224.802618029552 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.52573214617615 > 2.2845308787465597. IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -13.94699586344723 (T) = (0 -2592.7625435435334) / Math.Sqrt((4919.536191229149 / (32)) + (14450.998808596494 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.18084862529589368 = (2195.6773188382604 - 2592.7625435435334) / 2195.6773188382604 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Bag - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.Concurrent.IsEmpty(Int32).Bag(Size%3a%20512).html>) 46.02 ns 55.31 ns 1.20 0.65 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.Concurrent.IsEmpty<Int32>.Bag(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 55.31339062338936 > 50.430543768541376. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -4.010250998350441 (T) = (0 -56.00902489524304) / Math.Sqrt((20.5846307653969 / (32)) + (29.086145886795244 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1115770780520157 = (50.38699160061496 - 56.00902489524304) / 50.38699160061496 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Next_int_unseeded - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Random.Next_int_unseeded.html>) 31.54 ns 36.32 ns 1.15 0.68 False
[Next_int - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Random.Next_int.html>) 31.08 ns 39.53 ns 1.27 0.67 False
[Next_long - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Random.Next_long.html>) 92.72 ns 101.65 ns 1.10 0.34 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Random.Next_int_unseeded ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 36.31676164278097 > 32.450065407653106. IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -7.761320066892196 (T) = (0 -36.77698501590825) / Math.Sqrt((4.1244260360284954 / (32)) + (8.120823073221315 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.17499110542221658 = (31.299798650554855 - 36.77698501590825) / 31.299798650554855 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Random.Next_int ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 39.527610167892746 > 34.15863407003011. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -8.133106537691837 (T) = (0 -35.36393237750524) / Math.Sqrt((5.645423626696244 / (32)) + (6.777199158960745 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1905864814717677 = (29.702951383917444 - 35.36393237750524) / 29.702951383917444 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Random.Next_long ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 101.64717876401252 > 97.65172483793113. IsChangePoint: Marked as a change because one of 1/18/2023 3:58:17 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -7.849730294836891 (T) = (0 -102.9104190573666) / Math.Sqrt((9.193992187457193 / (31)) + (15.929321628870577 / (23))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (23) - 2, .025) and -0.08209034779825782 = (95.10335182894816 - 102.9104190573666) / 95.10335182894816 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in PerfLabTests.DictionaryExpansion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ExpandDictionaries - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.DictionaryExpansion.ExpandDictionaries.html>) 1.54 μs 1.78 μs 1.16 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.DictionaryExpansion*'
### Payloads [Baseline]() [Compare]() ### Histogram #### PerfLabTests.DictionaryExpansion.ExpandDictionaries ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.7846032902529425 > 1.619365508920234. IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -19.62660129151795 (T) = (0 -1764.8116436097232) / Math.Sqrt((671.913963459265 / (32)) + (2087.395903005932 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1359958826298745 = (1553.5370071272757 - 1764.8116436097232) / 1553.5370071272757 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Tests.Perf_HashCode

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Add - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_HashCode.Add.html>) 7.47 μs 10.35 μs 1.38 0.20 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_HashCode*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_HashCode.Add ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 10.349680475694125 > 7.89002947445516. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -31.23939466955221 (T) = (0 -9753.146522303574) / Math.Sqrt((121458.9151308833 / (32)) + (33038.97876432847 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.30399245155123705 = (7479.450138458374 - 9753.146522303574) / 7479.450138458374 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IndexOfString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.IndexOfString(input%3a%20%22Hello%20WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello%20WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello%20Worldbbbbbbbbbbbbbbbbbbb.html>) 22.90 ns 25.00 ns 1.09 0.35 False
[IndexOfString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.IndexOfString(input%3a%20%22More%20Test%27s%22%2c%20value%3a%20%22Tests%22%2c%20comparisonType%3a%20OrdinalIgnoreCase).html>) 35.30 ns 38.85 ns 1.10 0.14 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 25.00403969413373 > 24.177767610981515. IsChangePoint: Marked as a change because one of 11/25/2022 2:35:48 PM, 11/29/2022 2:55:01 PM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -9.687398929716887 (T) = (0 -25.477541967302102) / Math.Sqrt((0.3685764148622194 / (32)) + (0.789809588333082 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.09027316135075057 = (23.36803552582888 - 25.477541967302102) / 23.36803552582888 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.ReadOnlySpan.IndexOfString(input: "More Test's", value: "Tests", comparisonType: OrdinalIgnoreCase) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 38.850588015845304 > 36.9651005876322. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -15.641623283983641 (T) = (0 -39.1282291797092) / Math.Sqrt((0.5829229680748688 / (32)) + (0.9041020864067462 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.10785641882454225 = (35.318863089880395 - 39.1282291797092) / 35.318863089880395 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadLine - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange%3a%20%5b%20%20%200%2c%20%20%20%200%5d).html>) 303.55 μs 335.79 μs 1.11 0.31 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 335.7872339539007 > 312.892674150515. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM. IsRegressionStdDev: Marked as regression because -9.85741844919905 (T) = (0 -343558.87114841363) / Math.Sqrt((219339388.61183 / (32)) + (251492743.8565413 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1398517496621594 = (301406.62700236327 - 343558.87114841363) / 301406.62700236327 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

This is probably https://github.com/dotnet/runtime/pull/80668 as well