search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 20 Improvements on 7/31/2023 7:57:50 PM #20464

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 242242947c01615b243e8b4d9ee163030a4ceb47
Compare 865d02eeeed50f1c5ba300d6ed27aa645e195983
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.32 μs 664.77 ns 0.50 0.10 True
1.33 μs 655.90 ns 0.49 0.06 True
1.66 μs 952.58 ns 0.57 0.09 True
1.35 μs 652.26 ns 0.48 0.04 True
1.62 μs 963.40 ns 0.59 0.07 True
11.74 μs 11.05 μs 0.94 0.17 False
1.38 μs 656.75 ns 0.48 0.08 True
1.32 μs 659.24 ns 0.50 0.07 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 664.7677796630929 < 1.2639043336747082. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 261.3316313335103 (T) = (0 -658.3464469943999) / Math.Sqrt((1059.719514819917 / (299)) + (17.40598966939944 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.5127036965495165 = (1351.0187586746135 - 658.3464469943999) / 1351.0187586746135 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 655.9015741014947 < 1.2710965028198193. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 360.7953796928939 (T) = (0 -656.2888500240791) / Math.Sqrt((638.2676456484526 / (299)) + (9.056344497048254 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.5120626336950129 = (1345.026832017336 - 656.2888500240791) / 1345.026832017336 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 952.57507594455 < 1.5672012168928187. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 233.78798169439514 (T) = (0 -955.4000190719107) / Math.Sqrt((1667.836527910469 / (299)) + (15.272411093554556 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.4108777529500566 = (1621.7347483584604 - 955.4000190719107) / 1621.7347483584604 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 652.2618065007798 < 1.2865676119951253. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 321.6206323890243 (T) = (0 -658.057453443258) / Math.Sqrt((515.5905890657971 / (299)) + (17.025069239263672 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.5107357191358758 = (1344.9938595170208 - 658.057453443258) / 1344.9938595170208 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 963.3988447181206 < 1.560924444926643. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 222.99883117496205 (T) = (0 -954.8107699574962) / Math.Sqrt((1395.8553758902844 / (299)) + (25.874073129640806 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.41173332793897577 = (1623.091729151109 - 954.8107699574962) / 1623.091729151109 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, True)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 11.046320007763132 < 11.326392303596078. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 11.993633135489704 (T) = (0 -11146.268668964281) / Math.Sqrt((102741.11961187705 / (299)) + (31346.418086030848 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.0743242422282845 = (12041.223479585933 - 11146.268668964281) / 12041.223479585933 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, None, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 656.7520365098392 < 1.2649371916903016. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 304.03574879476827 (T) = (0 -659.8640216743138) / Math.Sqrt((705.6344196912044 / (299)) + (16.540893476430348 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.5103424804676682 = (1347.6031621132763 - 659.8640216743138) / 1347.6031621132763 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 659.235020979955 < 1.2677829255301776. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 234.64599132071098 (T) = (0 -663.2611024625329) / Math.Sqrt((749.3704364247502 / (299)) + (35.917716690548595 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.5076252575021534 = (1347.065649829578 - 663.2611024625329) / 1347.065649829578 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 242242947c01615b243e8b4d9ee163030a4ceb47
Compare 865d02eeeed50f1c5ba300d6ed27aa645e195983
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.37 μs 668.40 ns 0.28 0.16 True
2.18 μs 908.13 ns 0.42 0.14 True
1.78 μs 946.92 ns 0.53 0.15 True
5.47 μs 2.12 μs 0.39 0.13 True
  • [EscapeDataString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_Uri.EscapeDataString(input%3a%20%22%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b%7b.html>)
  • 📝 - Benchmark Source
  • ADX - Test Multi Config Graph
 15.79 μs 13.82 μs 0.88 0.03 False
3.77 μs 1.30 μs 0.35 0.11 True
  • [EscapeDataString - Duration of single invocation](<https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu 18.04_AOT=true_CompilationMode=wasm_RunKind=micro/System.Tests.Perf_Uri.EscapeDataString(input%3a%20%22a%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bca%7b%c3%bc.html>)
  • 📝 - Benchmark Source
  • ADX - Test Multi Config Graph
 39.89 μs 27.94 μs 0.70 0.27 False
5.41 μs 2.35 μs 0.43 0.08 True

graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'
### Payloads [Baseline]() [Compare]() ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 668.4004711751662 < 2.1736250201767877. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 260.1705822778606 (T) = (0 -686.3840697099646) / Math.Sqrt((7250.024565091985 / (299)) + (86.84363541207236 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.7022630835984867 = (2305.3374704275525 - 686.3840697099646) / 2305.3374704275525 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://dot.net/path/with?key=value#fragment") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 908.1268881040053 < 2.086215416026684. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 200.03567616543222 (T) = (0 -913.9911986092856) / Math.Sqrt((4544.560799037446 / (299)) + (159.5712510297472 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.5859035022159259 = (2207.1937422805154 - 913.9911986092856) / 2207.1937422805154 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path%20with%20escapable%20values?key=va%20lue") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 946.9168438469426 < 1.6827129619402361. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 115.94881262257616 (T) = (0 -971.0579524176223) / Math.Sqrt((3881.94002344322 / (289)) + (248.66352231329165 / (6))) is greater than 1.9680934526838334 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (289) + (6) - 2, .975) and 0.469364148658003 = (1829.9893419598056 - 971.0579524176223) / 1829.9893419598056 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=ünicode") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.115969872969549 < 4.898430649816938. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 141.0599132536531 (T) = (0 -2127.989213818599) / Math.Sqrt((27067.704209225638 / (287)) + (2477.2852354935403 / (6))) is greater than 1.9681495542477299 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (287) + (6) - 2, .975) and 0.5988553891452707 = (5304.793224778558 - 2127.989213818599) / 5304.793224778558 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 13.824074946601332 < 14.960596002861726. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 48.78662606268664 (T) = (0 -13709.016407165032) / Math.Sqrt((50349.71003773084 / (299)) + (7545.886324926865 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.127215993902215 = (15707.226887048502 - 13709.016407165032) / 15707.226887048502 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=va lue") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 1.3022249036447422 < 3.577335484940293. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 296.93531114070555 (T) = (0 -1294.0022707454607) / Math.Sqrt((15701.269391888078 / (287)) + (134.1518988028851 / (6))) is greater than 1.9681495542477299 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (287) + (6) - 2, .975) and 0.6682667085997022 = (3900.7308108368497 - 1294.0022707454607) / 3900.7308108368497 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.EscapeDataString(input: "a{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{ü") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 27.94300625961785 < 35.35294173402181. IsChangePoint: Marked as a change because one of 6/9/2023 10:45:21 AM, 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 44.730113287566176 (T) = (0 -27891.7486860093) / Math.Sqrt((8770993.314086666 / (299)) + (109685.25721928042 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.25922807758002736 = (37652.27574351337 - 27891.7486860093) / 37652.27574351337 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path%20with%20escapable%20values?key=%C3%BCnicode") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.3451875674789364 < 5.108520814744074. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 144.9564934962813 (T) = (0 -2285.9199617781037) / Math.Sqrt((48042.84349379679 / (286)) + (1919.1177993953631 / (6))) is greater than 1.9681778964020553 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (286) + (6) - 2, .975) and 0.583436904555549 = (5487.5719591509815 - 2285.9199617781037) / 5487.5719591509815 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 242242947c01615b243e8b4d9ee163030a4ceb47
Compare 865d02eeeed50f1c5ba300d6ed27aa645e195983
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
891.14 ns 88.32 ns 0.10 0.18 True
249.05 ns 64.21 ns 0.26 0.18 True
520.27 ns 334.30 ns 0.64 0.05 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'
### Payloads [Baseline]() [Compare]() ### System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 88.31907843852206 < 890.5867988936977. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 308.8549489447488 (T) = (0 -79.95144542862853) / Math.Sqrt((810.2207213890588 / (299)) + (27.786617471544286 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9127895344846659 = (916.7643465286948 - 79.95144542862853) / 916.7643465286948 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 64.20622405535393 < 239.10751548406725. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 103.81435259525033 (T) = (0 -65.73851715146522) / Math.Sqrt((77.46054203176247 / (256)) + (18.250869416895632 / (6))) is greater than 1.9691300033599934 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (256) + (6) - 2, .975) and 0.7427978626325316 = (255.59086648468892 - 65.73851715146522) / 255.59086648468892 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: InvariantCultureIgnoreCase) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 334.3049460436113 < 495.2607975101314. IsChangePoint: Marked as a change because one of 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 156.43426697874997 (T) = (0 -330.97559637970625) / Math.Sqrt((139.7417208194554 / (254)) + (4.744034070795882 / (6))) is greater than 1.9692013862338849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (6) - 2, .975) and 0.3537120760744222 = (512.1178721232291 - 330.97559637970625) / 512.1178721232291 is greater than 0.05. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 242242947c01615b243e8b4d9ee163030a4ceb47
Compare 865d02eeeed50f1c5ba300d6ed27aa645e195983
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Improvements in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.90 μs 2.98 μs 0.61 0.17 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*'
### Payloads [Baseline]() [Compare]() ### System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString: "http://name5", uriCount: 10) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small. IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small. IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline. IsImprovementWindowed: Marked as improvement because 2.9756114507993905 < 4.616084111727972. IsChangePoint: Marked as a change because one of 5/4/2023 10:11:14 AM, 7/31/2023 11:21:23 AM, 8/3/2023 7:53:20 AM falls between 7/25/2023 7:04:35 PM and 8/3/2023 7:53:20 AM. IsImprovementStdDev: Marked as improvement because 54.6217567002559 (T) = (0 -2934.7790797887196) / Math.Sqrt((25213.270576332063 / (299)) + (7580.083574754115 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.40591274119945403 = (4939.979836823968 - 2934.7790797887196) / 4939.979836823968 is greater than 0.05. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
lewing commented 1 year ago

This is https://github.com/dotnet/runtime/pull/89688

thanks @radekdoulik for tracking it down and @vargaz for fixing it