search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 28 Regressions on 9/25/2023 9:33:39 AM #22381

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
17.79 μs 20.79 μs 1.17 0.22 False
101.48 μs 137.48 μs 1.35 0.20 True
36.89 μs 39.71 μs 1.08 0.24 False
749.89 μs 803.10 μs 1.07 0.14 False
1.29 ms 1.60 ms 1.23 0.10 False
1.24 ms 1.36 ms 1.10 0.08 False
86.06 μs 95.26 μs 1.11 0.11 False
781.42 μs 873.38 μs 1.12 0.09 False
1.50 ms 1.67 ms 1.11 0.08 False
1.31 ms 1.64 ms 1.25 0.07 True
802.54 μs 868.84 μs 1.08 0.09 False

graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.CtorFromCollection<String>.ConcurrentQueue(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 20.787978261873963 > 18.965317098717193. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -6.446285455839081 (T) = (0 -19872.163341029598) / Math.Sqrt((1009105.0256527982 / (42)) + (362255.9881152697 / (13))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (13) - 2, .025) and -0.07979120534882721 = (18403.70920099306 - 19872.163341029598) / 18403.70920099306 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.SortedDictionaryDeepCopy(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 137.48314904129793 > 108.94030533112577. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -42.92070138953736 (T) = (0 -139267.60871832346) / Math.Sqrt((11104893.554786343 / (48)) + (1769204.2808393228 / (4))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (4) - 2, .025) and -0.33859863171507976 = (104039.85587516014 - 139267.60871832346) / 104039.85587516014 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.HashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 39.71139222249269 > 38.98248053043366. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -7.962062177849979 (T) = (0 -40079.72240509081) / Math.Sqrt((1630366.6176241364 / (40)) + (1124966.882407029 / (13))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (13) - 2, .025) and -0.07628401791718077 = (37238.98314745292 - 40079.72240509081) / 37238.98314745292 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.SortedSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 803.1015651785714 > 774.9388234070618. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -12.967498342046463 (T) = (0 -816363.2488836333) / Math.Sqrt((341217503.1795721 / (40)) + (255938721.77352625 / (14))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (14) - 2, .025) and -0.0896212736125049 = (749217.4287099605 - 816363.2488836333) / 749217.4287099605 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.ImmutableDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.595017863839286 > 1.3781096372721358. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -15.691307187595966 (T) = (0 -1563576.231361607) / Math.Sqrt((686677865.4854947 / (51)) + (912313327.1637554 / (4))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (4) - 2, .025) and -0.1847899248421085 = (1319707.5688924158 - 1563576.231361607) / 1319707.5688924158 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.SortedDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.3619214727564102 > 1.30618629609375. IsChangePoint: Marked as a change because one of 8/22/2023 10:50:09 AM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -7.88128601216427 (T) = (0 -1340714.7721619217) / Math.Sqrt((750891796.6656 / (39)) + (1217283864.6301596 / (14))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (14) - 2, .025) and -0.06448640223359557 = (1259494.5030286158 - 1340714.7721619217) / 1259494.5030286158 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.ConcurrentDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 95.2610811151501 > 89.49576909158087. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -19.342832167937136 (T) = (0 -96412.77510428181) / Math.Sqrt((3066342.621016605 / (38)) + (2996366.2185885916 / (14))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (14) - 2, .025) and -0.1222280714335974 = (85911.92606786132 - 96412.77510428181) / 85911.92606786132 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.SortedList(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 873.3829370039683 > 838.557670453125. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -15.06577253297324 (T) = (0 -872145.5064361242) / Math.Sqrt((393812162.5878681 / (41)) + (205232135.54331893 / (14))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (14) - 2, .025) and -0.09300616899501404 = (797932.8307341902 - 872145.5064361242) / 797932.8307341902 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.ImmutableSortedDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.6721954384920632 > 1.5844709500994318. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/25/2023 7:14:09 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -7.214024956654312 (T) = (0 -1660636.367060486) / Math.Sqrt((1177140865.4189398 / (43)) + (2857490311.9864154 / (9))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (9) - 2, .025) and -0.08775340097832193 = (1526666.2145730043 - 1660636.367060486) / 1526666.2145730043 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.ImmutableHashSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.6387112558333339 > 1.376886542983774. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -21.414162869327882 (T) = (0 -1669285.7609706959) / Math.Sqrt((675973992.0282217 / (49)) + (1038846727.0926137 / (4))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (4) - 2, .025) and -0.2692863189041258 = (1315137.2831402775 - 1669285.7609706959) / 1315137.2831402775 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.CtorFromCollection<String>.ImmutableSortedSet(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 868.8407432870368 > 832.3640935312502. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -9.278120759169903 (T) = (0 -862885.1894981649) / Math.Sqrt((280013356.244697 / (41)) + (549251423.3253015 / (13))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (13) - 2, .025) and -0.08146487385369934 = (797885.5442834254 - 862885.1894981649) / 797885.5442834254 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
111.43 μs 136.88 μs 1.23 0.27 False
122.57 μs 129.84 μs 1.06 0.14 False
163.61 μs 247.21 μs 1.51 0.09 False
417.09 μs 590.77 μs 1.42 0.19 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.TryGetValueTrue<Int32, Int32>.ImmutableSortedDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 136.8760234649123 > 113.23153133561642. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -6.355425199879271 (T) = (0 -121644.58071586123) / Math.Sqrt((11761360.577509208 / (42)) + (41279015.95672719 / (13))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (13) - 2, .025) and -0.10756378311783676 = (109830.7678257832 - 121644.58071586123) / 109830.7678257832 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.TryGetValueTrue<Int32, Int32>.ImmutableDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 129.84263881944443 > 129.0388271033654. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -12.047133680411038 (T) = (0 -132199.0248603856) / Math.Sqrt((15396507.147803498 / (42)) + (4725239.025149531 / (12))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (12) - 2, .025) and -0.0863220671977734 = (121694.13551673506 - 132199.0248603856) / 121694.13551673506 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.TryGetValueTrue<Int32, Int32>.SortedList(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 247.20976484374998 > 172.7855142298177. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -27.001101375442005 (T) = (0 -242320.0133499313) / Math.Sqrt((17415092.40754853 / (50)) + (23075431.23331388 / (3))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (3) - 2, .025) and -0.4618890468935838 = (165758.14277071512 - 242320.0133499313) / 165758.14277071512 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.TryGetValueTrue<Int32, Int32>.SortedDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 590.7692908950617 > 437.9371644657768. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -24.6834455568363 (T) = (0 -578798.1682145097) / Math.Sqrt((155983296.8950696 / (51)) + (151439608.50758883 / (4))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (4) - 2, .025) and -0.3751298887196894 = (420904.3618078857 - 578798.1682145097) / 420904.3618078857 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.47 μs 6.29 μs 1.15 0.13 False
6.09 μs 7.53 μs 1.24 0.07 False
4.63 μs 5.13 μs 1.11 0.11 False
4.73 μs 6.10 μs 1.29 0.11 True
5.44 μs 6.45 μs 1.19 0.12 False

graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromReader(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.290029006142222 > 5.783902418972616. IsChangePoint: Marked as a change because one of 8/24/2023 4:35:49 AM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -6.279759191902675 (T) = (0 -5977.909047334134) / Math.Sqrt((16980.450094073287 / (40)) + (71885.62138630949 / (13))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (13) - 2, .025) and -0.08820985733584802 = (5493.342122418586 - 5977.909047334134) / 5493.342122418586 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromReader(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 7.531121888462339 > 6.770510054938634. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -28.20274264344306 (T) = (0 -7555.397125830107) / Math.Sqrt((29015.48896729044 / (49)) + (5162.177629212597 / (4))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (4) - 2, .025) and -0.1932692388474911 = (6331.678450981794 - 7555.397125830107) / 6331.678450981794 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromString(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 5.1270241606120655 > 4.876458907957442. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -5.973773711503677 (T) = (0 -4866.964379372049) / Math.Sqrt((11932.405937949508 / (37)) + (50044.00661276606 / (14))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (14) - 2, .025) and -0.08298138084036241 = (4494.042525085172 - 4866.964379372049) / 4494.042525085172 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.098351387245233 > 5.039170687904072. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -48.181265197517035 (T) = (0 -6097.87073736232) / Math.Sqrt((14291.975393003304 / (50)) + (1667.4666928844208 / (4))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (4) - 2, .025) and -0.26494715118778905 = (4820.652571640168 - 6097.87073736232) / 4820.652571640168 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.446518936319363 > 5.5057696851495646. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -14.475760555193064 (T) = (0 -6723.484777141117) / Math.Sqrt((21146.60146250967 / (51)) + (38354.23635958714 / (4))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (4) - 2, .025) and -0.27443064850531723 = (5275.6772485239435 - 6723.484777141117) / 5275.6772485239435 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Sort<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
769.18 μs 842.46 μs 1.10 0.21 False
766.35 μs 819.10 μs 1.07 0.19 False
638.81 μs 703.37 μs 1.10 0.13 False

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;String&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Sort<String>.Array_ComparerStruct(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 842.4649999999999 > 778.9180928272693. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -9.928559519194533 (T) = (0 -834709.4936569344) / Math.Sqrt((600211846.5482801 / (42)) + (311667142.2248639 / (13))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (13) - 2, .025) and -0.07942260569067018 = (773292.581845499 - 834709.4936569344) / 773292.581845499 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Sort<String>.Array_Comparison(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 819.1021537846154 > 797.5148301120001. IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -10.814911853113378 (T) = (0 -834251.1283538461) / Math.Sqrt((449960461.5077634 / (41)) + (365088510.7928223 / (14))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (14) - 2, .025) and -0.08567072841196764 = (768420.0250789869 - 834251.1283538461) / 768420.0250789869 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Sort<String>.List(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 703.365861323077 > 671.4420738829998. IsChangePoint: Marked as a change because one of 8/22/2023 4:48:44 PM, 9/24/2023 11:15:45 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -11.458042990214961 (T) = (0 -712738.134122232) / Math.Sqrt((226411884.0466894 / (39)) + (294491628.33820295 / (14))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (14) - 2, .025) and -0.09085370942606955 = (653376.4591562187 - 712738.134122232) / 653376.4591562187 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.75 ms 1.92 ms 1.10 0.08 False
1.70 ms 2.20 ms 1.29 0.04 True
1.73 ms 2.12 ms 1.23 0.03 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*'
### Payloads [Baseline]() [Compare]() ### MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>.JsonNet_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.9184603889508929 > 1.8519109622005208. IsChangePoint: Marked as a change because one of 8/24/2023 11:06:19 AM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -7.374565828469329 (T) = (0 -1879325.2529064822) / Math.Sqrt((566146260.0348606 / (42)) + (2338384231.653212 / (13))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (13) - 2, .025) and -0.05771396042250733 = (1776780.2290855455 - 1879325.2529064822) / 1776780.2290855455 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>.SystemTextJson_Reflection_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.19617635840708 > 1.7957160063859419. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -10.212561736518968 (T) = (0 -2243126.201253955) / Math.Sqrt((941053754.1879385 / (50)) + (10416694220.424978 / (4))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (4) - 2, .025) and -0.30407640728381763 = (1720088.0168716705 - 2243126.201253955) / 1720088.0168716705 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>.SystemTextJson_SourceGen_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.12133618019943 > 1.8096704399275463. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -24.041581948143243 (T) = (0 -2157619.6982138082) / Math.Sqrt((1035082635.1295816 / (49)) + (1340877194.414458 / (4))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (4) - 2, .025) and -0.26636831146707973 = (1703785.287957987 - 2157619.6982138082) / 1703785.287957987 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in Benchstone.BenchF.Simpsn

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
0.00 ns 2.71 ns ∞ 0.82 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchF.Simpsn*'
### Payloads [Baseline]() [Compare]() ### Benchstone.BenchF.Simpsn.Test #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.7137281710269074 > 0. IsChangePoint: Marked as a change because one of 8/26/2023 7:53:22 PM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -5.224587285892773 (T) = (0 -1.5726299900502736) / Math.Sqrt((0.11118871947841251 / (39)) + (0.7906038862684537 / (13))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (13) - 2, .025) and -5.182967741299115 = (0.25434872958270444 - 1.5726299900502736) / 0.25434872958270444 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline c8f97e8df6eaa4f21944ed80661c089403cf8f85
Compare af8172dc268dcaae71bfda421e88c98386c527c9
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.13 μs 8.81 μs 1.08 0.15 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'
### Payloads [Baseline]() [Compare]() ### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, True)) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 8.814052313456298 > 8.485652732163391. IsChangePoint: Marked as a change because one of 8/25/2023 8:22:05 AM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM. IsRegressionStdDev: Marked as regression because -9.846889548748587 (T) = (0 -8950.579115506443) / Math.Sqrt((34516.05877277441 / (41)) + (61604.29876248326 / (13))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (13) - 2, .025) and -0.08954366337656115 = (8214.979735431678 - 8950.579115506443) / 8214.979735431678 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)