search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 17 Improvements on 11/8/2022 11:25:18 AM #9821

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2aebbf75167b3aa21cae23e6fd702688dd724663
Compare d0d21fdb83aa7287e2141395ebecd50ae1cc851e
Diff Diff

Improvements in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[ToString4 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Version.ToString4.html>) 187.19 ns 157.00 ns 0.84 0.25 False
[TryFormat4 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Version.TryFormat4.html>) 113.71 ns 84.76 ns 0.75 0.34 False
[TryFormatL - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Version.TryFormatL.html>) 125.88 ns 101.45 ns 0.81 0.17 False
[TryFormat2 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Version.TryFormat2.html>) 62.44 ns 50.18 ns 0.80 0.40 False

graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Version*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_Version.ToString4 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 157.0023409130927 < 174.79324897401037. IsChangePoint: Marked as a change because one of 10/3/2022 1:11:02 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 7.213453346219343 (T) = (0 -165.89624104881693) / Math.Sqrt((163.2512097974344 / (299)) + (45.301750771034605 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.055701078467901496 = (175.68191307436308 - 165.89624104881693) / 175.68191307436308 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Version.TryFormat4 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 84.75531794327121 < 104.40072856091592. IsChangePoint: Marked as a change because one of 10/3/2022 1:11:02 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 2.418561458583292 (T) = (0 -89.57435436064883) / Math.Sqrt((111.53130835859905 / (299)) + (167.59979664134718 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.059305098918213936 = (95.2214732509335 - 89.57435436064883) / 95.2214732509335 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Version.TryFormatL ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 101.44886883354673 < 117.70272255969459. IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 23.31916051493527 (T) = (0 -104.47026089143917) / Math.Sqrt((132.90774112343345 / (299)) + (47.74219070810103 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.23087823993558093 = (135.83058797177847 - 104.47026089143917) / 135.83058797177847 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Version.TryFormat2 ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 50.17684300518864 < 59.6810435061321. IsChangePoint: Marked as a change because one of 10/5/2022 11:49:14 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 3.7245139092361015 (T) = (0 -53.391664295252056) / Math.Sqrt((28.20389215405272 / (299)) + (36.631782498820236 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.06934544047295439 = (57.370013125370015 - 53.391664295252056) / 57.370013125370015 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2aebbf75167b3aa21cae23e6fd702688dd724663
Compare d0d21fdb83aa7287e2141395ebecd50ae1cc851e
Diff Diff

Improvements in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TrailingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong.html>) 1.10 μs 392.97 ns 0.36 0.01 False
[LeadingZeroCount_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong.html>) 5.51 μs 391.60 ns 0.07 0.27 False
[LeadingZeroCount_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint.html>) 4.95 μs 398.78 ns 0.08 0.18 False
[Log2_uint - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.Log2_uint.html>) 5.19 μs 562.70 ns 0.11 0.22 False
[Log2_ulong - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Numerics.Tests.Perf_BitOperations.Log2_ulong.html>) 5.03 μs 660.71 ns 0.13 0.22 False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 240.8928656415084 (T) = (0 -392.9805210879774) / Math.Sqrt((2563.0783135792526 / (299)) + (3.9491122323751227 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.6436842306567075 = (1102.8996045060255 - 392.9805210879774) / 1102.8996045060255 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 391.5974882632125 < 4.977160864147067. IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 271.4152916095214 (T) = (0 -393.17815031205595) / Math.Sqrt((94874.10364012644 / (299)) + (3.836409071972573 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.92480486958438 = (5228.771439571598 - 393.17815031205595) / 5228.771439571598 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 383.00611525386796 (T) = (0 -396.80474969999153) / Math.Sqrt((44504.27194981358 / (299)) + (5.239734790942085 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.921767279195421 = (5072.107241306207 - 396.80474969999153) / 5072.107241306207 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.Log2_uint ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 308.8313404595154 (T) = (0 -562.9290120153591) / Math.Sqrt((60900.466902258064 / (299)) + (5.1902773907573065 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.8867828666083517 = (4972.118575622628 - 562.9290120153591) / 4972.118575622628 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Numerics.Tests.Perf_BitOperations.Log2_ulong ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 660.711403077679 < 4.770810325276962. IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 278.21195339135335 (T) = (0 -662.939231748595) / Math.Sqrt((77731.53782249373 / (299)) + (7.262396642204559 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.8712868777744467 = (5150.517835989394 - 662.939231748595) / 5150.517835989394 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2aebbf75167b3aa21cae23e6fd702688dd724663
Compare d0d21fdb83aa7287e2141395ebecd50ae1cc851e
Diff Diff

Improvements in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryFormat - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Int32.TryFormat(value%3a%204).html>) 24.91 ns 20.79 ns 0.83 0.66 False
[TryFormat - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Int32.TryFormat(value%3a%2012345).html>) 30.03 ns 23.87 ns 0.79 0.62 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_Int32.TryFormat(value: 4) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking. IsChangePoint: Marked as a change because one of 10/3/2022 5:29:39 AM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 4.5751854858485235 (T) = (0 -22.513372808607446) / Math.Sqrt((8.639848659795868 / (299)) + (5.3071433936784285 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.07948275945151365 = (24.457307062704167 - 22.513372808607446) / 24.457307062704167 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_Int32.TryFormat(value: 12345) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 23.87074482532943 < 26.441063909362644. IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 6.234221471837572 (T) = (0 -25.571258039484004) / Math.Sqrt((5.994794806247272 / (299)) + (7.668303891742147 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.10652294607276493 = (28.61993816974569 - 25.571258039484004) / 28.61993816974569 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2aebbf75167b3aa21cae23e6fd702688dd724663
Compare d0d21fdb83aa7287e2141395ebecd50ae1cc851e
Diff Diff

Improvements in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[FormatterInt64 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value%3a%2012345).html>) 21.52 ns 17.28 ns 0.80 0.54 False
[FormatterUInt32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value%3a%204294967295).html>) 29.66 ns 22.30 ns 0.75 0.44 False
[FormatterUInt32 - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value%3a%2012345).html>) 21.12 ns 17.17 ns 0.81 0.50 False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 17.28447460099655 < 20.099848922182655. IsChangePoint: Marked as a change because one of 10/3/2022 1:11:02 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 8.650364309413359 (T) = (0 -17.702874474828434) / Math.Sqrt((4.921032152529315 / (299)) + (1.2973647881826997 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.101568162710342 = (19.70419317311098 - 17.702874474828434) / 19.70419317311098 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 22.299279063193975 < 24.379440285632963. IsChangePoint: Marked as a change because one of 10/11/2022 10:45:23 PM, 11/8/2022 11:25:18 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 5.7763118210275834 (T) = (0 -23.527065649342443) / Math.Sqrt((5.372073702189478 / (299)) + (2.244043461543509 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.06641894980391316 = (25.200881749260958 - 23.527065649342443) / 25.200881749260958 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 12345) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 17.17012377447551 < 20.43963298681604. IsChangePoint: Marked as a change because one of 10/3/2022 1:11:02 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 3.994912705133847 (T) = (0 -18.25507601051631) / Math.Sqrt((6.3979167306604 / (299)) + (2.2800190414774337 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.06048381285026945 = (19.430294294234443 - 18.25507601051631) / 19.430294294234443 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2aebbf75167b3aa21cae23e6fd702688dd724663
Compare d0d21fdb83aa7287e2141395ebecd50ae1cc851e
Diff Diff

Improvements in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryFormat - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_UInt32.TryFormat(value%3a%204294967295).html>) 28.62 ns 26.29 ns 0.92 0.54 False
[TryFormat - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_UInt32.TryFormat(value%3a%200).html>) 27.20 ns 22.43 ns 0.82 0.58 False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_UInt32.TryFormat(value: 4294967295) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 26.285518380663863 < 30.281284304876966. IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 16.42510461744144 (T) = (0 -27.78558725442382) / Math.Sqrt((14.344573282566714 / (299)) + (3.983211974972786 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.1936879753418125 = (34.460092873106674 - 27.78558725442382) / 34.460092873106674 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ```#### System.Tests.Perf_UInt32.TryFormat(value: 0) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 22.429239968204666 < 24.6673483737704. IsChangePoint: Marked as a change because one of 10/3/2022 1:11:02 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 3.690467771914795 (T) = (0 -22.545320387119165) / Math.Sqrt((6.551960064234543 / (299)) + (7.368415281244762 / (35))) is greater than 1.9671350567190904 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (35) - 2, .975) and 0.07314604817295696 = (24.324566284339767 - 22.545320387119165) / 24.324566284339767 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2aebbf75167b3aa21cae23e6fd702688dd724663
Compare d0d21fdb83aa7287e2141395ebecd50ae1cc851e
Diff Diff

Improvements in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
[TryFormat - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompliationMode=wasm_RunKind=micro/System.Tests.Perf_Int64.TryFormat(value%3a%2012345).html>) 31.74 ns 25.88 ns 0.82 0.42 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int64*'

Related Issues

Regressions

### Payloads [Baseline]() [Compare]() ### Histogram ### Edge Detector Info [Collection Data]() #### System.Tests.Perf_Int64.TryFormat(value: 12345) ```log ``` ### Description of detection logic ```IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed:Marked as improvement because 25.881479065628916 < 30.25141672932004. IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 10/13/2022 10:51:15 PM, 11/8/2022 6:56:37 AM, 11/14/2022 7:25:10 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 7:25:10 PM. IsImprovementStdDev: Marked as improvement because 11.84694886992585 (T) = (0 -26.87957683127487) / Math.Sqrt((5.518310151218476 / (299)) + (2.5915756986698435 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.12069364018343817 = (30.56906905220429 - 26.87957683127487) / 30.56906905220429 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

@radekdoulik this looks like https://github.com/dotnet/runtime/pull/77777