search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.26k stars 4.73k forks source link

[Perf] Linux/x64: 27 Regressions on 8/17/2022 6:09:21 PM #74395

Closed performanceautofiler[bot] closed 2 years ago

performanceautofiler[bot] commented 2 years ago

There are some large wasm regressions here; as much as a 10 x slow down. Unfortunately, there seems to be some kind of glitch in the reporting, and the narrowest I can get the change range is this: https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...a250bfe05688352da06c11cbfce33550b388a868.

@lewing @radical @pavelsavara Any likely candidates in that date range?

Automated issue reporting below.

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Uri_IsNotMatch - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options%3a%20None).html>) 444.95 ns 592.31 ns 1.33 0.06 False
[Uri_IsMatch - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options%3a%20IgnoreCase%2c%20Compiled).html>) 455.72 ns 505.99 ns 1.11 0.05 False
[Backtracking - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options%3a%20Compiled).html>) 2.45 μs 2.93 μs 1.19 0.10 False
[Backtracking - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options%3a%20None).html>) 2.45 μs 2.91 μs 1.19 0.10 False
[Uri_IsNotMatch - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options%3a%20IgnoreCase%2c%20Compiled).html>) 446.25 ns 581.71 ns 1.30 0.05 False
[Uri_IsNotMatch - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options%3a%20Compiled).html>) 447.94 ns 593.46 ns 1.32 0.04 False

graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 592.3093690327008 > 465.23264235782176. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -39.47621223602034 (T) = (0 -594.5364016454982) / Math.Sqrt((78.92038015570309 / (14)) + (73.27831635677934 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.32709221691484786 = (447.99931313563445 - 594.5364016454982) / 447.99931313563445 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 505.9861520887883 > 482.7014638466213. IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/25/2022 6:06:34 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -14.329767862712716 (T) = (0 -514.4415933301957) / Math.Sqrt((16.440001607480827 / (14)) + (123.32550738351799 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.12037177742305741 = (459.17043225905917 - 514.4415933301957) / 459.17043225905917 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9266354356553093 > 2.591927353654952. IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -50.14154390540031 (T) = (0 -2934.2504087143566) / Math.Sqrt((556.432351718883 / (14)) + (417.26555557386655 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18845364681210353 = (2468.964958444245 - 2934.2504087143566) / 2468.964958444245 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9101414399726724 > 2.5780127447204717. IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/25/2022 2:40:01 AM, 6/29/2022 4:06:39 AM, 7/12/2022 12:16:14 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -61.70631510275718 (T) = (0 -2923.812224534253) / Math.Sqrt((407.1945204507192 / (14)) + (236.0815117136678 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18619294374645873 = (2464.870694053968 - 2923.812224534253) / 2464.870694053968 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 581.7136123036038 > 471.5091607783785. IsChangePoint: Marked as a change because one of 6/25/2022 5:15:47 AM, 6/26/2022 6:11:06 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -65.94725454140149 (T) = (0 -592.9345145576187) / Math.Sqrt((24.613985181656737 / (14)) + (26.562821064295214 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3181628970020087 = (449.81884705310074 - 592.9345145576187) / 449.81884705310074 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 593.4594763018782 > 471.43001759852376. IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/24/2022 6:48:53 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -48.543170908408 (T) = (0 -596.0567739968475) / Math.Sqrt((34.7262092388468 / (14)) + (56.62787404726851 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3178926565327463 = (452.2802149645615 - 596.0567739968475) / 452.2802149645615 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadSpanEmptyLoop - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact%3a%20True%2c%20TestCase%3a%20Json40KB).html>) 92.65 μs 99.48 μs 1.07 0.09 False
[ReadMultiSpanSequenceEmptyLoop - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact%3a%20True%2c%20TestCase%3a%20Json400B).html>) 1.70 μs 1.91 μs 1.13 0.07 False
[ReadSpanEmptyLoop - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact%3a%20False%2c%20TestCase%3a%20LotsOfNumbers).html>) 4.25 μs 4.95 μs 1.16 0.09 False
[ReadMultiSpanSequenceEmptyLoop - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact%3a%20True%2c%20TestCase%3a%20Json4KB).html>) 11.52 μs 12.59 μs 1.09 0.12 False
[ReadMultiSpanSequenceEmptyLoop - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact%3a%20True%2c%20TestCase%3a%20Json40KB).html>) 112.87 μs 118.86 μs 1.05 0.07 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Reader*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 99.47909983507735 > 97.49529907633277. IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -9.875672080579108 (T) = (0 -101229.28427961336) / Math.Sqrt((265137.5448871564 / (14)) + (5976092.35420695 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.08769210576340399 = (93067.95897775218 - 101229.28427961336) / 93067.95897775218 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.90914647431983 > 1.777974693072054. IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -24.12292592652499 (T) = (0 -1927.1512790997767) / Math.Sqrt((611.4742780301284 / (14)) + (471.64347753333664 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.1398571366423715 = (1690.6954539728583 - 1927.1512790997767) / 1690.6954539728583 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfNumbers) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.946905717995979 > 4.460535892543025. IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/25/2022 5:15:47 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -14.376124247128779 (T) = (0 -4757.429574447925) / Math.Sqrt((4758.814727171611 / (14)) + (8231.09975483114 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.11985725605871154 = (4248.246415968665 - 4757.429574447925) / 4248.246415968665 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.58755040658602 > 12.38535184349092. IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -6.466117999484596 (T) = (0 -12591.240550799725) / Math.Sqrt((69596.82198162597 / (14)) + (79394.98687585884 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06418279234267017 = (11831.840019778581 - 12591.240550799725) / 11831.840019778581 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 118.8626892992424 > 118.55053428507193. IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -7.506023118864004 (T) = (0 -121572.12886191734) / Math.Sqrt((4192742.5510894423 / (14)) + (6482122.752315378 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06649265512776761 = (113992.4670623754 - 121572.12886191734) / 113992.4670623754 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Format_MultipleArgs - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_String.Format_MultipleArgs.html>) 529.02 ns 850.06 ns 1.61 0.14 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_String.Format_MultipleArgs ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 850.0557408302284 > 547.6817332098481. IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -86.40554379355675 (T) = (0 -848.1242802325862) / Math.Sqrt((88.87602574889881 / (14)) + (69.21223243326581 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.6173858106947179 = (524.3796963127123 - 848.1242802325862) / 524.3796963127123 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ExpandEnvironmentVariables - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Environment.ExpandEnvironmentVariables.html>) 382.06 ns 1.06 μs 2.76 0.11 False
[SetEnvironmentVariable - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Environment.SetEnvironmentVariable.html>) 194.79 ns 661.73 ns 3.40 0.16 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Environment.ExpandEnvironmentVariables ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.05539799053797 > 404.12774324999464. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -100.66329094480135 (T) = (0 -1058.1419394441955) / Math.Sqrt((68.03089949492843 / (14)) + (358.51775965172686 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -1.7472293516620847 = (385.16694603747425 - 1058.1419394441955) / 385.16694603747425 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Environment.SetEnvironmentVariable ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 661.725814258165 > 213.09369271099766. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -164.42300266168527 (T) = (0 -647.4959303229203) / Math.Sqrt((49.2398659816711 / (14)) + (35.11775717258779 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -2.2433183359130027 = (199.6399561379005 - 647.4959303229203) / 199.6399561379005 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IndexOfString - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.ReadOnlySpan.IndexOfString(input%3a%20%22StrIng%22%2c%20value%3a%20%22string%22%2c%20comparisonType%3a%20OrdinalIgnoreCase).html>) 24.67 ns 36.76 ns 1.49 0.21 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 36.76482912607041 > 26.19735640294146. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -67.5375793028699 (T) = (0 -36.38052307472043) / Math.Sqrt((0.037995025208353364 / (14)) + (0.23599627187678154 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.4615325383200185 = (24.892037721266604 - 36.38052307472043) / 24.892037721266604 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ICollection - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.ContainsFalse(Int32).ICollection(Size%3a%20512).html>) 186.65 μs 2.42 ms 12.95 0.01 False
[Stack - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.ContainsFalse(Int32).Stack(Size%3a%20512).html>) 148.17 μs 2.98 ms 20.11 0.01 False
[Queue - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.ContainsFalse(Int32).Queue(Size%3a%20512).html>) 186.46 μs 2.42 ms 12.98 0.02 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'
### Histogram #### System.Collections.ContainsFalse<Int32>.ICollection(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.4180025503826528 > 196.43006513020836. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -373.14299335258266 (T) = (0 -2427206.700892857) / Math.Sqrt((1083311.6941798208 / (14)) + (323590191.29539 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -11.95486324505923 = (187358.7281454749 - 2427206.700892857) / 187358.7281454749 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.ContainsFalse<Int32>.Stack(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9792487988095235 > 155.34581797022406. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -42.64814865820476 (T) = (0 -2532821.408927845) / Math.Sqrt((874624.8610895702 / (14)) + (28136948852.7768 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.092038104366907 = (148187.20818792962 - 2532821.408927845) / 148187.20818792962 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.ContainsFalse<Int32>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.4195034313186814 > 195.68580234160373. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -91.92109792877557 (T) = (0 -2470159.070670352) / Math.Sqrt((714329.7028732462 / (14)) + (5552447103.319803 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -12.216076602459486 = (186905.62600179392 - 2470159.070670352) / 186905.62600179392 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[Load - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName%3a%20%22repeated.xml%22).html>) 277.65 μs 294.38 μs 1.06 0.09 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*'
### Payloads [Baseline]() [Compare]() ### Histogram #### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 294.3815035812672 > 286.28722408634746. IsChangePoint: Marked as a change because one of 7/1/2022 12:02:05 PM, 7/10/2022 5:59:30 PM, 7/18/2022 3:18:54 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -9.519909182959154 (T) = (0 -300647.38905054185) / Math.Sqrt((42979330.934151694 / (14)) + (35851042.04585498 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.09181732773423353 = (275364.18539395486 - 300647.38905054185) / 275364.18539395486 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 2 years ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Segment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ReadSingleSegmentSequenceByN - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes%3a%208192%2c%20TestCase%3a%20Json40KB).html>) 117.72 μs 124.68 μs 1.06 0.05 False
[ReadSingleSegmentSequenceByN - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes%3a%204096%2c%20TestCase%3a%20Json4KB).html>) 11.91 μs 12.64 μs 1.06 0.04 False
[ReadSingleSegmentSequenceByN - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes%3a%208192%2c%20TestCase%3a%20Json4KB).html>) 11.86 μs 13.09 μs 1.10 0.04 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Segment*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json40KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 124.68098554894179 > 124.27589552531899. IsChangePoint: Marked as a change because one of 6/11/2022 8:34:59 AM, 7/19/2022 2:55:40 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -5.581969937441441 (T) = (0 -127368.57600209878) / Math.Sqrt((7023383.564901864 / (14)) + (15611587.1916289 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.07013381276822936 = (119021.16771043884 - 127368.57600209878) / 119021.16771043884 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 4096, TestCase: Json4KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.640910305240409 > 12.515637436028303. IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -10.206712835881275 (T) = (0 -12774.988873320122) / Math.Sqrt((20535.81036398369 / (14)) + (46387.504846041105 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06953141358175953 = (11944.472795369229 - 12774.988873320122) / 11944.472795369229 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json4KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 13.090807536867004 > 12.469607141755468. IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -12.446003189221445 (T) = (0 -12721.135652115448) / Math.Sqrt((3972.5173419869543 / (14)) + (40342.44528186691 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.07243763434320451 = (11861.888509634673 - 12721.135652115448) / 11861.888509634673 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[IndexOfValue - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).IndexOfValue(Size%3a%20512).html>) 138.67 ns 2.42 μs 17.43 0.01 False
[IndexOfAnyTwoValues - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).IndexOfAnyTwoValues(Size%3a%20512).html>) 200.08 ns 3.04 μs 15.18 0.01 False
[IndexOfAnyThreeValues - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Memory.Span(Byte).IndexOfAnyThreeValues(Size%3a%20512).html>) 276.67 ns 3.75 μs 13.54 0.01 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.Span<Byte>.IndexOfValue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.4170771770847206 > 146.37516945438924. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -14.608891603462073 (T) = (0 -2611.6152103241934) / Math.Sqrt((259.65556211673896 / (14)) + (254663.71967576197 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.02608588753459 = (153.38905415931507 - 2611.6152103241934) / 153.38905415931507 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.0370345882066276 > 209.74119752252065. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -70.32494865522497 (T) = (0 -3138.38731756492) / Math.Sqrt((0.1937329534100386 / (14)) + (15711.882878990908 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -14.689264738200135 = (200.0340595900324 - 3138.38731756492) / 200.0340595900324 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.7468551530143617 > 290.8288738091121. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -23.271230281894688 (T) = (0 -3929.8603884016356) / Math.Sqrt((1.2682850554058953 / (14)) + (221737.7219606583 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -13.181022917520329 = (277.12108014058583 - 3929.8603884016356) / 277.12108014058583 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[GetFileNameWithoutExtension - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.IO.Tests.Perf_Path.GetFileNameWithoutExtension.html>) 109.03 ns 153.38 ns 1.41 0.25 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.IO.Tests.Perf_Path.GetFileNameWithoutExtension ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 153.38430839699873 > 114.74546136153367. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -27.261989122675345 (T) = (0 -151.75101640427368) / Math.Sqrt((12.744288495830018 / (14)) + (13.286249350442171 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.384142762437261 = (109.63537903927168 - 151.75101640427368) / 109.63537903927168 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ImmutableArray - Duration of single invocation](<https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Collections.ContainsTrue(Int32).ImmutableArray(Size%3a%20512).html>) 125.83 μs 1.25 ms 9.90 0.07 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Collections.ContainsTrue<Int32>.ImmutableArray(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.245679729474852 > 132.27599546370968. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -100.87925648694225 (T) = (0 -1265867.2470181838) / Math.Sqrt((1331132.729081719 / (14)) + (1147780671.5517871 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -9.02935874958368 = (126216.16980953344 - 1265867.2470181838) / 126216.16980953344 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
naricc commented 2 years ago

Something in this date range: https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...a250bfe05688352da06c11cbfce33550b388a868

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 2 years ago

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details
There are some large wasm regressions here; as much as a 10 x slow down. Unfortunately, there seems to be some kind of glitch in the reporting, and the narrowest I can get the change range is this: https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...a250bfe05688352da06c11cbfce33550b388a868. @lewing @radical @pavelsavara Any likely candidates in that date range? Automated issue reporting below. ### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Uri_IsNotMatch - Duration of single invocation]() | 444.95 ns | 592.31 ns | 1.33 | 0.06 | False | | | [Uri_IsMatch - Duration of single invocation]() | 455.72 ns | 505.99 ns | 1.11 | 0.05 | False | | | [Backtracking - Duration of single invocation]() | 2.45 μs | 2.93 μs | 1.19 | 0.10 | False | | | [Backtracking - Duration of single invocation]() | 2.45 μs | 2.91 μs | 1.19 | 0.10 | False | | | [Uri_IsNotMatch - Duration of single invocation]() | 446.25 ns | 581.71 ns | 1.30 | 0.05 | False | | | [Uri_IsNotMatch - Duration of single invocation]() | 447.94 ns | 593.46 ns | 1.32 | 0.04 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 592.3093690327008 > 465.23264235782176. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -39.47621223602034 (T) = (0 -594.5364016454982) / Math.Sqrt((78.92038015570309 / (14)) + (73.27831635677934 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.32709221691484786 = (447.99931313563445 - 594.5364016454982) / 447.99931313563445 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 505.9861520887883 > 482.7014638466213. IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/25/2022 6:06:34 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -14.329767862712716 (T) = (0 -514.4415933301957) / Math.Sqrt((16.440001607480827 / (14)) + (123.32550738351799 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.12037177742305741 = (459.17043225905917 - 514.4415933301957) / 459.17043225905917 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9266354356553093 > 2.591927353654952. IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -50.14154390540031 (T) = (0 -2934.2504087143566) / Math.Sqrt((556.432351718883 / (14)) + (417.26555557386655 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18845364681210353 = (2468.964958444245 - 2934.2504087143566) / 2468.964958444245 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9101414399726724 > 2.5780127447204717. IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/25/2022 2:40:01 AM, 6/29/2022 4:06:39 AM, 7/12/2022 12:16:14 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -61.70631510275718 (T) = (0 -2923.812224534253) / Math.Sqrt((407.1945204507192 / (14)) + (236.0815117136678 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18619294374645873 = (2464.870694053968 - 2923.812224534253) / 2464.870694053968 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 581.7136123036038 > 471.5091607783785. IsChangePoint: Marked as a change because one of 6/25/2022 5:15:47 AM, 6/26/2022 6:11:06 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -65.94725454140149 (T) = (0 -592.9345145576187) / Math.Sqrt((24.613985181656737 / (14)) + (26.562821064295214 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3181628970020087 = (449.81884705310074 - 592.9345145576187) / 449.81884705310074 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 593.4594763018782 > 471.43001759852376. IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/24/2022 6:48:53 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -48.543170908408 (T) = (0 -596.0567739968475) / Math.Sqrt((34.7262092388468 / (14)) + (56.62787404726851 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3178926565327463 = (452.2802149645615 - 596.0567739968475) / 452.2802149645615 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in System.Text.Json.Tests.Perf_Reader Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ReadSpanEmptyLoop - Duration of single invocation]() | 92.65 μs | 99.48 μs | 1.07 | 0.09 | False | | | [ReadMultiSpanSequenceEmptyLoop - Duration of single invocation]() | 1.70 μs | 1.91 μs | 1.13 | 0.07 | False | | | [ReadSpanEmptyLoop - Duration of single invocation]() | 4.25 μs | 4.95 μs | 1.16 | 0.09 | False | | | [ReadMultiSpanSequenceEmptyLoop - Duration of single invocation]() | 11.52 μs | 12.59 μs | 1.09 | 0.12 | False | | | [ReadMultiSpanSequenceEmptyLoop - Duration of single invocation]() | 112.87 μs | 118.86 μs | 1.05 | 0.07 | False | | | ![graph]() ![graph]() ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Reader*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 99.47909983507735 > 97.49529907633277. IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -9.875672080579108 (T) = (0 -101229.28427961336) / Math.Sqrt((265137.5448871564 / (14)) + (5976092.35420695 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.08769210576340399 = (93067.95897775218 - 101229.28427961336) / 93067.95897775218 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.90914647431983 > 1.777974693072054. IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -24.12292592652499 (T) = (0 -1927.1512790997767) / Math.Sqrt((611.4742780301284 / (14)) + (471.64347753333664 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.1398571366423715 = (1690.6954539728583 - 1927.1512790997767) / 1690.6954539728583 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfNumbers) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.946905717995979 > 4.460535892543025. IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/25/2022 5:15:47 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -14.376124247128779 (T) = (0 -4757.429574447925) / Math.Sqrt((4758.814727171611 / (14)) + (8231.09975483114 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.11985725605871154 = (4248.246415968665 - 4757.429574447925) / 4248.246415968665 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.58755040658602 > 12.38535184349092. IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -6.466117999484596 (T) = (0 -12591.240550799725) / Math.Sqrt((69596.82198162597 / (14)) + (79394.98687585884 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06418279234267017 = (11831.840019778581 - 12591.240550799725) / 11831.840019778581 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 118.8626892992424 > 118.55053428507193. IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -7.506023118864004 (T) = (0 -121572.12886191734) / Math.Sqrt((4192742.5510894423 / (14)) + (6482122.752315378 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06649265512776761 = (113992.4670623754 - 121572.12886191734) / 113992.4670623754 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in System.Tests.Perf_String Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Format_MultipleArgs - Duration of single invocation]() | 529.02 ns | 850.06 ns | 1.61 | 0.14 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_String.Format_MultipleArgs ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 850.0557408302284 > 547.6817332098481. IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -86.40554379355675 (T) = (0 -848.1242802325862) / Math.Sqrt((88.87602574889881 / (14)) + (69.21223243326581 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.6173858106947179 = (524.3796963127123 - 848.1242802325862) / 524.3796963127123 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in System.Tests.Perf_Environment Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ExpandEnvironmentVariables - Duration of single invocation]() | 382.06 ns | 1.06 μs | 2.76 | 0.11 | False | | | [SetEnvironmentVariable - Duration of single invocation]() | 194.79 ns | 661.73 ns | 3.40 | 0.16 | False | | | ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Tests.Perf_Environment.ExpandEnvironmentVariables ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.05539799053797 > 404.12774324999464. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -100.66329094480135 (T) = (0 -1058.1419394441955) / Math.Sqrt((68.03089949492843 / (14)) + (358.51775965172686 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -1.7472293516620847 = (385.16694603747425 - 1058.1419394441955) / 385.16694603747425 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Environment.SetEnvironmentVariable ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 661.725814258165 > 213.09369271099766. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -164.42300266168527 (T) = (0 -647.4959303229203) / Math.Sqrt((49.2398659816711 / (14)) + (35.11775717258779 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -2.2433183359130027 = (199.6399561379005 - 647.4959303229203) / 199.6399561379005 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in System.Memory.ReadOnlySpan Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [IndexOfString - Duration of single invocation]() | 24.67 ns | 36.76 ns | 1.49 | 0.21 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 36.76482912607041 > 26.19735640294146. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -67.5375793028699 (T) = (0 -36.38052307472043) / Math.Sqrt((0.037995025208353364 / (14)) + (0.23599627187678154 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.4615325383200185 = (24.892037721266604 - 36.38052307472043) / 24.892037721266604 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in System.Collections.ContainsFalse<Int32> Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [ICollection - Duration of single invocation]() | 186.65 μs | 2.42 ms | 12.95 | 0.01 | False | | | [Stack - Duration of single invocation]() | 148.17 μs | 2.98 ms | 20.11 | 0.01 | False | | | [Queue - Duration of single invocation]() | 186.46 μs | 2.42 ms | 12.98 | 0.02 | False | | | ![graph]() ![graph]() ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse<Int32>*' ```
### Histogram #### System.Collections.ContainsFalse<Int32>.ICollection(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.4180025503826528 > 196.43006513020836. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -373.14299335258266 (T) = (0 -2427206.700892857) / Math.Sqrt((1083311.6941798208 / (14)) + (323590191.29539 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -11.95486324505923 = (187358.7281454749 - 2427206.700892857) / 187358.7281454749 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.ContainsFalse<Int32>.Stack(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.9792487988095235 > 155.34581797022406. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -42.64814865820476 (T) = (0 -2532821.408927845) / Math.Sqrt((874624.8610895702 / (14)) + (28136948852.7768 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.092038104366907 = (148187.20818792962 - 2532821.408927845) / 148187.20818792962 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Collections.ContainsFalse<Int32>.Queue(Size: 512) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.4195034313186814 > 195.68580234160373. IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -91.92109792877557 (T) = (0 -2470159.070670352) / Math.Sqrt((714329.7028732462 / (14)) + (5552447103.319803 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -12.216076602459486 = (186905.62600179392 - 2470159.070670352) / 186905.62600179392 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
### Run Information Architecture | x64 -- | -- OS | ubuntu 18.04 Baseline | [543bcc5ee7d6a2b9471b016770227421c43a756e](https://github.com/dotnet/runtime/commit/543bcc5ee7d6a2b9471b016770227421c43a756e) Compare | [f142128e89b63577a9bbba7e2b760ec82102a7a9](https://github.com/dotnet/runtime/commit/f142128e89b63577a9bbba7e2b760ec82102a7a9) Diff | [Diff](https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...f142128e89b63577a9bbba7e2b760ec82102a7a9) ### Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [Load - Duration of single invocation]() | 277.65 μs | 294.38 μs | 1.06 | 0.09 | False | | | ![graph]() [Test Report]() ### Repro ```cmd git clone https://github.com/dotnet/performance.git python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' ```
### Payloads [Baseline]() [Compare]() ### Histogram #### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml") ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 294.3815035812672 > 286.28722408634746. IsChangePoint: Marked as a change because one of 7/1/2022 12:02:05 PM, 7/10/2022 5:59:30 PM, 7/18/2022 3:18:54 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM. IsRegressionStdDev: Marked as regression because -9.519909182959154 (T) = (0 -300647.38905054185) / Math.Sqrt((42979330.934151694 / (14)) + (35851042.04585498 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.09181732773423353 = (275364.18539395486 - 300647.38905054185) / 275364.18539395486 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
Author: performanceautofiler[bot]
Assignees: naricc
Labels: `arch-wasm`, `untriaged`, `area-Codegen-AOT-mono`, `refs/heads/main`, `ubuntu 18.04`, `RunKind=micro`, `PGO`, `Regression`, `x64`, `AOT=true`, `WASM`, `CompliationMode=wasm`
Milestone: -
lewing commented 2 years ago

The span indexof regressions are pretty brutal

cc @marek-safar @vargaz

lewing commented 2 years ago

@adamsitnik we think this might be related to the Vector128 changes

lewing commented 2 years ago

Also regressed the mono interpreter pretty badly https://github.com/dotnet/perf-autofiling-issues/issues/7976

cc @BrzVlad @vargaz

lewing commented 2 years ago

we can see the impact of https://github.com/dotnet/runtime/pull/73768 very clearly in other aot json tests but the span here is large in perf pipeline and the other vectorization might have caused issues as well.

vargaz commented 2 years ago

The usage of generic structs like:

private static int IndexOfAnyValueType<TValue, TNegator>(ref TValue searchSpace, TValue value0, TValue value1, TValue value2, TValue value3, int length)
            where TValue : struct, INumber<TValue>
            where TNegator : struct, INegator<TValue>

probably causes wasm to fall back to the interpreter for these methods, since the AOT compiler cannot determine the instantiations that will be used at runtime.

Imho such code should be avoided in low level BCL code, its better to have some code duplication.

tannergooding commented 2 years ago

Imho such code should be avoided in low level BCL code, its better to have some code duplication.

This isn't really practical, IMO, for us or our customers who are going to do the same. We have many features that are built on and around generics and generic specialization.

Much like with Vector<T>, Vector64<T>, Vector128<T>, and Vector256<T> this really seems like something that WASM and AOT will have to handle as they are such core concepts to .NET as a whole (especially in perf oriented code).

Many of these algorithms are very long and very complex. It's not just a small amount of duplication, but large amounts of duplication in some of our most critical code (where having one place to review/update is often core/critical due to its complexity).

tannergooding commented 2 years ago

Notably, in the case of IndexOfAnyValueType, these are all "internal" generic methods that are called from other methods where the type is concrete and well-known.

WASM/AOT could recognize the pattern of if (typeof(T) == typeof(...)) for value types and/or the cases where we are calling with concrete type information and either specialize or inline the methods to ensure it doesn't need to interpret (e.g. https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs,1162 has a size check and then always calls the API as IndexOfAnyValueType<byte>).

lewing commented 2 years ago

in the mean time string handling is now slower than net6 in many cases and it happened post the rc1 branch

lewing commented 2 years ago

cc @danmoseley

https://github.com/dotnet/runtime/pull/74086

tannergooding commented 2 years ago

Notably in WASM. Most other targets, including Arm64, have gotten significantly faster (often 2x or more).

The “best” fix for .NET 7 given that and the short timeframe is likely to ifdef for WASM or similar. I don’t think reverting it is the right choice given the gains we see on almost all our other targets.

For .NET 8 I think this really needs a more in depth look from the WASM side on how generics can be better supported. That could be one of the suggestions above or something as “simple” as some attribute that tells WASM/AOT what to specialize for in an opt-in manner. I don’t believe duplicating code or maintaining the status quo is viable, especially given the prevalence of genetics in .NET and all the newer features and existing perf patterns that encourage users to write generic and generic specialized algorithms

danmoseley commented 2 years ago

Historically when doing this kind of vectorization work we have assumed it's reasonable to measure only on CoreCLR x64 and Arm64 and I assume that's what @adamsitnik did. Maybe it's fine to rely on the perf lab to catch anything Mono specific - it's just that timeframes are tighter at the moment.

If #if is feasible that seems reasonable as me in the short term?

jkotas commented 2 years ago

the AOT compiler cannot determine the instantiations that will be used at runtime.

@vargaz Why is the AOT compiler not able to determine the instantiations that will be used at runtime in this case?

The instantiation chain for these generic helper methods always uses a small set of concrete types. There is no ambiguity about what this will be instantiated with.

vargaz commented 2 years ago

So what happens is that the AOT compiler creates a shared instantiation for bytes and enums whose base type is bytes, and that ends up being called, and then it cannot inline the call to DontNegate.NegateIfNeeded () into that. This appears fixable, but not in the net7 timeframe.

adamsitnik commented 2 years ago

This appears fixable, but not in the net7 timeframe.

@jkotas @vargaz should I port the fix to just release/7.0-rc2 branch or both main and release/7.0-rc2?

vargaz commented 2 years ago

I think both if possible.

stephentoub commented 2 years ago

We should start with a fix in main and then backport it to rc2.

lewing commented 2 years ago

resolved for net7 in https://github.com/dotnet/runtime/pull/75996