search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 10 Regressions on 9/19/2022 9:42:49 AM #8716

Open performanceautofiler[bot] opened 2 years ago

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 774324e59288db97067bbd1918af3371ff8a8ce4
Compare 5d4e4320a9196e4ab8ccc068e7e32222f43c7a59
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IsSuffix_SecondHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options%3a%20(en-US%2c%20IgnoreCase%2c%20False)).html>) 7.02 μs 9.15 μs 1.30 0.05 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(en-US%2c%20None%2c%20False)).html>) 7.03 μs 9.18 μs 1.31 0.07 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(en-US%2c%20IgnoreCase%2c%20False)).html>) 7.18 μs 9.37 μs 1.30 0.08 False
[IsSuffix_SecondHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options%3a%20(%2c%20IgnoreCase%2c%20False)).html>) 7.05 μs 9.18 μs 1.30 0.07 False
[IsSuffix_SecondHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options%3a%20(en-US%2c%20None%2c%20False)).html>) 7.03 μs 9.14 μs 1.30 0.04 False
[IsSuffix_SecondHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options%3a%20(%2c%20None%2c%20False)).html>) 7.03 μs 9.33 μs 1.33 0.06 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(en-US%2c%20IgnoreNonSpace%2c%20False)).html>) 7.01 μs 9.13 μs 1.30 0.09 False
[IsSuffix_SecondHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options%3a%20(en-US%2c%20IgnoreNonSpace%2c%20False)).html>) 7.01 μs 9.18 μs 1.31 0.07 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(%2c%20IgnoreCase%2c%20False)).html>) 7.02 μs 9.24 μs 1.32 0.06 False
[IsPrefix_FirstHalf - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_CompliationMode=wasm_RunKind=micro/System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options%3a%20(%2c%20None%2c%20False)).html>) 8.60 μs 9.21 μs 1.07 0.06 False

graph graph graph graph graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.146219169921054 > 7.397206523372493. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -72.14459678850908 (T) = (0 -9134.165414101746) / Math.Sqrt((5577.901337299989 / (15)) + (18756.840986604013 / (43))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (43) - 2, .025) and -0.28952669258024355 = (7083.347298399062 - 9134.165414101746) / 7083.347298399062 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.178662743082375 > 7.372136169391424. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -9.233594828888066 (T) = (0 -9158.669851770159) / Math.Sqrt((591123.1047939324 / (15)) + (38565.588217286044 / (43))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (43) - 2, .025) and -0.2537669990922901 = (7304.921774461211 - 9158.669851770159) / 7304.921774461211 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.372905340658239 > 7.464663183100074. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -13.489607128506247 (T) = (0 -9150.095533127565) / Math.Sqrt((281548.0420563815 / (15)) + (112147.63076317271 / (43))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (43) - 2, .025) and -0.2747855356726573 = (7177.752866719968 - 9150.095533127565) / 7177.752866719968 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.18348551803603 > 7.421940476000295. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -27.651822710159305 (T) = (0 -9157.689036145186) / Math.Sqrt((68415.07240637708 / (14)) + (13984.317709605248 / (43))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (43) - 2, .025) and -0.2787591609499386 = (7161.38684734216 - 9157.689036145186) / 7161.38684734216 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.136538239601641 > 7.386159820682434. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -128.40916660949605 (T) = (0 -9115.442521779267) / Math.Sqrt((591.3631204651656 / (15)) + (9619.831532576654 / (43))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (43) - 2, .025) and -0.2961997132563986 = (7032.436767694424 - 9115.442521779267) / 7032.436767694424 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.333394786135692 > 7.394028508778206. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -7.8886814424997995 (T) = (0 -9119.195532754666) / Math.Sqrt((789030.5655424793 / (15)) + (10712.97012747765 / (43))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (43) - 2, .025) and -0.24824110862229634 = (7305.636282736808 - 9119.195532754666) / 7305.636282736808 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.127812827678047 > 7.373965953627256. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -5.781051762321992 (T) = (0 -9139.418919427428) / Math.Sqrt((1010784.8804273771 / (15)) + (38755.79396925026 / (41))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (41) - 2, .025) and -0.19810300936295924 = (7628.241351540322 - 9139.418919427428) / 7628.241351540322 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreNonSpace, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.180665480409528 > 7.394706275213567. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 1:55:13 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -16.02603665647612 (T) = (0 -9145.50793063352) / Math.Sqrt((202303.60377368957 / (14)) + (40096.06620470023 / (43))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (43) - 2, .025) and -0.27769014808944326 = (7157.84491592816 - 9145.50793063352) / 7157.84491592816 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.235020208456826 > 7.3765861020476455. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -12.867445942226679 (T) = (0 -9113.971817941212) / Math.Sqrt((334679.8562379473 / (15)) + (8054.214604918475 / (43))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (43) - 2, .025) and -0.268668938444202 = (7183.8850481496665 - 9113.971817941212) / 7183.8850481496665 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False)) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 9.205651092494612 > 7.406669897296502. IsChangePoint: Marked as a change because one of 7/19/2022 10:33:10 AM, 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM. IsRegressionStdDev: Marked as regression because -7.197869081253489 (T) = (0 -9098.460669015802) / Math.Sqrt((729401.5237770944 / (14)) + (5725.756020785804 / (42))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (42) - 2, .025) and -0.2207182407247588 = (7453.366686495901 - 9098.460669015802) / 7453.366686495901 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
naricc commented 2 years ago

This issue is kind of at the borderline where I think it migh be a real regression. But the only possible change in the time period is this: https://github.com/dotnet/runtime/commit/16cb35fda05165d679603c1c9c7ca5710e9bd3df. I don't think that could have actually caused a regression, and there probably isn't anyhting to do about it if so. Thoughts @pavelsavara ?

performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 774324e59288db97067bbd1918af3371ff8a8ce4
Compare 5d4e4320a9196e4ab8ccc068e7e32222f43c7a59
Diff Diff

Regressions in Benchstone.BenchI.Midpoint

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Test - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Benchstone.BenchI.Midpoint.Test.html>) 446.06 ms 488.17 ms 1.09 0.00 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.Midpoint*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Benchstone.BenchI.Midpoint.Test ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 488.16992828571426 > 469.2849915330769. IsChangePoint: Marked as a change because one of 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -45.42245472521791 (T) = (0 -490120891.4223245) / Math.Sqrt((5735042970199.754 / (63)) + (52532420465809.555 / (64))) is less than -1.9791241094234997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (64) - 2, .025) and -0.09708931309095771 = (446746573.4776412 - 490120891.4223245) / 446746573.4776412 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 774324e59288db97067bbd1918af3371ff8a8ce4
Compare 5d4e4320a9196e4ab8ccc068e7e32222f43c7a59
Diff Diff

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IFooObjIsIFooInterAlia - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.CastingPerf.IFooObjIsIFooInterAlia.html>) 217.41 μs 260.80 μs 1.20 0.01 False
[IFooObjIsDescendantOfIFoo - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/PerfLabTests.CastingPerf.IFooObjIsDescendantOfIFoo.html>) 217.43 μs 260.78 μs 1.20 0.03 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### PerfLabTests.CastingPerf.IFooObjIsIFooInterAlia ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 260.79940572916666 > 228.29012969751602. IsChangePoint: Marked as a change because one of 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -26.158445788114452 (T) = (0 -260832.50263962176) / Math.Sqrt((112596992.26717064 / (63)) + (23605.92062185152 / (63))) is less than -1.9792801166046092 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (63) - 2, .025) and -0.1548513330917352 = (225858.07814875044 - 260832.50263962176) / 225858.07814875044 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### PerfLabTests.CastingPerf.IFooObjIsDescendantOfIFoo ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 260.7821300480769 > 228.29960635633677. IsChangePoint: Marked as a change because one of 9/7/2022 8:46:47 AM, 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -25.992640663745995 (T) = (0 -260880.39656736064) / Math.Sqrt((111946125.68745045 / (62)) + (54249.39808850449 / (64))) is less than -1.9792801166046092 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (64) - 2, .025) and -0.15461711054254035 = (225945.37547150688 - 260880.39656736064) / 225945.37547150688 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 774324e59288db97067bbd1918af3371ff8a8ce4
Compare 5d4e4320a9196e4ab8ccc068e7e32222f43c7a59
Diff Diff

Regressions in StoreBlock.LocalAddress

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[InitBlockAllOnes64 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/StoreBlock.LocalAddress.InitBlockAllOnes64.html>) 9.83 ns 10.99 ns 1.12 0.12 False
[InitBlockAllOnes32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/StoreBlock.LocalAddress.InitBlockAllOnes32.html>) 6.74 ns 8.30 ns 1.23 0.01 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'StoreBlock.LocalAddress*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### StoreBlock.LocalAddress.InitBlockAllOnes64 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 10.990465177432256 > 10.425049420787204. IsChangePoint: Marked as a change because one of 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -23.540196505217526 (T) = (0 -10.971108998695462) / Math.Sqrt((0.07201286488765231 / (63)) + (0.036147883983822834 / (64))) is less than -1.9791241094234997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (64) - 2, .025) and -0.09729994016533242 = (9.998277223128639 - 10.971108998695462) / 9.998277223128639 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### StoreBlock.LocalAddress.InitBlockAllOnes32 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 8.296628014389594 > 7.095197232993736. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -37.116381974581515 (T) = (0 -7.566274790365224) / Math.Sqrt((0.014234126847205882 / (63)) + (0.0120449169623512 / (63))) is less than -1.9792801166046092 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (63) - 2, .025) and -0.11134392032991318 = (6.8082207964201595 - 7.566274790365224) / 6.8082207964201595 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 774324e59288db97067bbd1918af3371ff8a8ce4
Compare 5d4e4320a9196e4ab8ccc068e7e32222f43c7a59
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IndexOfAnyFourValues - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).IndexOfAnyFourValues(Size%3a%20512).html>) 798.62 ns 853.86 ns 1.07 0.00 False
[Fill - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).Fill(Size%3a%20512).html>) 55.96 ns 60.34 ns 1.08 0.10 False
[IndexOfAnyFiveValues - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).IndexOfAnyFiveValues(Size%3a%20512).html>) 913.97 ns 966.57 ns 1.06 0.01 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 853.8582739403888 > 838.8169792965887. IsChangePoint: Marked as a change because one of 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -84.30689218154689 (T) = (0 -855.4498109746909) / Math.Sqrt((17.944943329811082 / (63)) + (12.01275995588891 / (64))) is less than -1.9791241094234997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (64) - 2, .025) and -0.07266976505099988 = (797.4959664627246 - 855.4498109746909) / 797.4959664627246 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.Fill(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 60.34429199097505 > 56.51732626564056. IsChangePoint: Marked as a change because one of 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -38.73157952496264 (T) = (0 -60.51824503815818) / Math.Sqrt((0.8123560808368875 / (62)) + (1.0744580193211364 / (63))) is less than -1.9794386850931271 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (63) - 2, .025) and -0.12503834079915388 = (53.792162314370515 - 60.51824503815818) / 53.792162314370515 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 966.573065709142 > 959.5215913258461. IsChangePoint: Marked as a change because one of 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -81.1656725582191 (T) = (0 -965.5165998077811) / Math.Sqrt((25.491811616901064 / (64)) + (2.24304059717725 / (63))) is less than -1.9791241094234997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (64) + (63) - 2, .025) and -0.058621206883975034 = (912.0510656023555 - 965.5165998077811) / 912.0510656023555 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 774324e59288db97067bbd1918af3371ff8a8ce4
Compare 5d4e4320a9196e4ab8ccc068e7e32222f43c7a59
Diff Diff

Regressions in Benchstone.BenchI.BenchE

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Test - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Benchstone.BenchI.BenchE.Test.html>) 599.72 ms 858.19 ms 1.43 0.01 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.BenchE*'
### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### Benchstone.BenchI.BenchE.Test ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionWindowed: Marked as regression because 858.1919996923077 > 629.4491835423078. IsChangePoint: Marked as a change because one of 9/19/2022 6:07:30 AM, 9/30/2022 12:37:41 PM falls between 8/31/2022 10:21:55 AM and 9/30/2022 12:37:41 PM. IsRegressionStdDev: Marked as regression because -404.52468543189013 (T) = (0 -858485526.1819928) / Math.Sqrt((12498082544908.828 / (64)) + (13362454618894.65 / (63))) is less than -1.9791241094234997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (64) + (63) - 2, .025) and -0.43011632927559595 = (600290695.6644889 - 858485526.1819928) / 600290695.6644889 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)