[Perf] Linux/x64: 7 Regressions on 2/20/2023 10:31:48 AM

[performanceautofiler[bot]]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	ce719aa21f73ff387956b2f2ab148d32a9adaed7
Compare	ac7afb9ccb88b895eeb3264e38ab22d0c5d726ec
Diff	Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
[ImmutableQueue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CreateAddAndClear(String).ImmutableQueue(Size%3a%20512).html>)	78.08 μs	84.52 μs	1.08	0.11	False
[List - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CreateAddAndClear(String).List(Size%3a%20512).html>)	26.84 μs	31.45 μs	1.17	0.11	False
[Array - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CreateAddAndClear(String).Array(Size%3a%20512).html>)	15.04 μs	19.48 μs	1.30	0.12	False
[ICollection - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CreateAddAndClear(String).ICollection(Size%3a%20512).html>)	37.61 μs	41.63 μs	1.11	0.08	False
[Queue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CreateAddAndClear(String).Queue(Size%3a%20512).html>)	55.65 μs	58.50 μs	1.05	0.04	False
[Stack - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CreateAddAndClear(String).Stack(Size%3a%20512).html>)	28.90 μs	34.38 μs	1.19	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear<String>*'

### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CreateAddAndClear<String>.ImmutableQueue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 84.51855509355508 > 82.95441342485351. IsChangePoint: Marked as a change because one of 12/20/2022 6:23:25 PM, 1/12/2023 10:41:19 PM, 2/20/2023 9:09:49 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -9.354869790854298 (T) = (0 -83857.72361250872) / Math.Sqrt((1021307.6941235112 / (18)) + (6749918.724358785 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.05834842341412919 = (79234.51460530536 - 83857.72361250872) / 79234.51460530536 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CreateAddAndClear<String>.List(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 31.450075326961766 > 28.30815486563112. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/22/2023 4:57:34 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -16.998535315687764 (T) = (0 -31464.69049424439) / Math.Sqrt((957374.3932902545 / (32)) + (992195.0662132638 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.17366586214838398 = (26808.899797637958 - 31464.69049424439) / 26808.899797637958 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CreateAddAndClear<String>.Array(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 19.481575171660424 > 15.495659061487354. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/22/2023 4:57:34 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -25.131093582822373 (T) = (0 -20113.493680066385) / Math.Sqrt((398248.9420151822 / (32)) + (736847.9093346691 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.3657482991451496 = (14727.086749920056 - 20113.493680066385) / 14727.086749920056 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CreateAddAndClear<String>.ICollection(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 41.629355987678096 > 38.29785371439163. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/22/2023 4:57:34 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -22.68983413519793 (T) = (0 -42403.44983192799) / Math.Sqrt((514208.79156203644 / (32)) + (1014028.3095236731 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.15394885271657072 = (36746.38588365839 - 42403.44983192799) / 36746.38588365839 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CreateAddAndClear<String>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 58.49501758962012 > 56.02047632052023. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/22/2023 4:57:34 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -26.737697491288408 (T) = (0 -59407.862429518784) / Math.Sqrt((967250.3996745312 / (32)) + (536209.2080108668 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1175256888409261 = (53160.176112940506 - 59407.862429518784) / 53160.176112940506 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.CreateAddAndClear<String>.Stack(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 34.37730942758166 > 29.296412790249597. IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/22/2023 4:57:34 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -21.85147456883403 (T) = (0 -33951.174503274626) / Math.Sqrt((358477.6789088243 / (32)) + (1303071.9123996997 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.20599658575924937 = (28151.965688941204 - 33951.174503274626) / 28151.965688941204 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	ce719aa21f73ff387956b2f2ab148d32a9adaed7
Compare	ac7afb9ccb88b895eeb3264e38ab22d0c5d726ec
Diff	Diff

Regressions in System.Collections.CtorFromCollection<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
[ImmutableQueue - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Collections.CtorFromCollection(String).ImmutableQueue(Size%3a%20512).html>)	33.29 μs	36.85 μs	1.11	0.18	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection<String>*'

### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.CtorFromCollection<String>.ImmutableQueue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 36.846786654874215 > 34.90741149447959. IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/20/2023 9:09:49 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM. IsRegressionStdDev: Marked as regression because -8.929213141884581 (T) = (0 -36356.0954424401) / Math.Sqrt((194735.38873573055 / (18)) + (3892525.819150494 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.09254365841976918 = (33276.560769227974 - 36356.0954424401) / 33276.560769227974 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

[radekdoulik]

@kg, looks like https://github.com/dotnet/runtime/pull/81417, some of it looks improved later by https://github.com/dotnet/runtime/pull/82757