search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Windows/x64: 18 Regressions on 7/28/2023 9:42:28 PM #20306

Open performanceautofiler[bot] opened 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 1da23b146496d95f017439a48f62e78ec15b0289
Compare 60c49c6156e20183e19b87dc7ba8e147674a9f85
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.91 ns 16.95 ns 1.55 0.11 False
10.91 ns 16.93 ns 1.55 0.09 False
14.20 ns 16.65 ns 1.17 0.14 False
14.20 ns 16.54 ns 1.16 0.11 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'
### Payloads [Baseline]() [Compare]() ### System.Text.Perf_Ascii.ToUpperInPlace_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.945390757000887 > 11.466947246222322. IsChangePoint: Marked as a change because one of 6/5/2023 10:43:22 AM, 6/9/2023 2:21:24 AM, 7/1/2023 6:36:20 PM, 7/4/2023 5:42:46 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -17.400310834690554 (T) = (0 -16.938327827161864) / Math.Sqrt((3.1446976062206784 / (32)) + (4.515776295810331E-05 / (8))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (8) - 2, .025) and -0.4750190555227033 = (11.483463731361368 - 16.938327827161864) / 11.483463731361368 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLowerInPlace_Bytes(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.93264212378833 > 11.469720154781747. IsChangePoint: Marked as a change because one of 6/5/2023 10:43:22 AM, 6/8/2023 12:15:18 PM, 7/1/2023 6:36:20 PM, 7/4/2023 5:42:46 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -26.071054912183286 (T) = (0 -16.935070328190182) / Math.Sqrt((1.4713638661739326 / (30)) + (1.679228413554539E-05 / (8))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (8) - 2, .025) and -0.517316787108403 = (11.161196180043499 - 16.935070328190182) / 11.161196180043499 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToLower_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.651865503386166 > 14.912934233197227. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/14/2023 2:26:40 PM, 7/28/2023 2:45:53 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -27.202120180911493 (T) = (0 -16.640526250180592) / Math.Sqrt((0.18884898562693675 / (32)) + (0.0034181048944267687 / (10))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (10) - 2, .025) and -0.14832292118584853 = (14.491155704700448 - 16.640526250180592) / 14.491155704700448 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Perf_Ascii.ToUpper_Bytes_Chars(Size: 128) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 16.53869617071014 > 14.910169461578253. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/14/2023 2:26:40 PM, 7/28/2023 2:45:53 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -53.50427174640943 (T) = (0 -16.635476646596025) / Math.Sqrt((0.03931862552165771 / (30)) + (0.0039032733867398147 / (10))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (10) - 2, .025) and -0.15293365485770946 = (14.428823875948968 - 16.635476646596025) / 14.428823875948968 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 1da23b146496d95f017439a48f62e78ec15b0289
Compare 60c49c6156e20183e19b87dc7ba8e147674a9f85
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.06 μs 1.20 μs 1.14 0.02 False
430.76 ns 464.40 ns 1.08 0.10 False
1.04 μs 1.15 μs 1.11 0.02 False
1.08 μs 1.19 μs 1.10 0.03 False
475.72 ns 512.89 ns 1.08 0.07 False
470.52 ns 512.56 ns 1.09 0.07 False
1.05 μs 1.15 μs 1.09 0.02 False
469.84 ns 516.46 ns 1.10 0.08 False
1.07 μs 1.19 μs 1.12 0.03 False
1.10 μs 1.23 μs 1.11 0.01 False
1.74 μs 1.86 μs 1.07 0.02 False
75.25 ns 94.87 ns 1.26 0.14 False
1.03 μs 1.13 μs 1.09 0.03 False

graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'
### Payloads [Baseline]() [Compare]() ### System.Reflection.Attributes.GetCustomAttributesMethodOverrideMiss #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.2033458442652154 > 1.0913220517782147. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -15.39883513932232 (T) = (0 -1165.7624038407716) / Math.Sqrt((173.38500082790478 / (34)) + (518.5609719690433 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.1241677024492517 = (1037.000441571926 - 1165.7624038407716) / 1037.000441571926 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.IsDefinedClassHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 464.4024663805764 > 463.01272814206584. IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 6/2/2023 6:53:36 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -12.955348776057459 (T) = (0 -475.12741264280453) / Math.Sqrt((45.129432289396675 / (30)) + (62.38506443276502 / (8))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (8) - 2, .025) and -0.09070812336848467 = (435.61371045394475 - 475.12741264280453) / 435.61371045394475 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesMethodOverrideHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1480937760970877 > 1.091773074982817. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -33.523913574800794 (T) = (0 -1154.5618440576136) / Math.Sqrt((178.79718759358343 / (34)) + (62.64067415391154 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.11737837883592961 = (1033.277416071377 - 1154.5618440576136) / 1033.277416071377 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesMethodBaseHitInherit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1886302245082816 > 1.1331919335256004. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -31.681420601244753 (T) = (0 -1195.6894419741193) / Math.Sqrt((217.49917016024926 / (34)) + (57.46150057783705 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.10820623371717317 = (1078.9412706726146 - 1195.6894419741193) / 1078.9412706726146 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.IsDefinedMethodOverrideMissInherit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 512.8928723718305 > 499.7785030160689. IsChangePoint: Marked as a change because one of 4/28/2023 4:10:17 AM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -8.373435036695229 (T) = (0 -516.1869485246174) / Math.Sqrt((121.02304060969952 / (32)) + (177.7481674621225 / (8))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (8) - 2, .025) and -0.09017442904358537 = (473.4902367664873 - 516.1869485246174) / 473.4902367664873 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.IsDefinedMethodOverrideHitInherit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 512.5642135231152 > 492.8083987425571. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 6/16/2023 8:17:59 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -12.062315454309987 (T) = (0 -506.5960195534238) / Math.Sqrt((139.67482281839293 / (29)) + (46.37094071963958 / (8))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (8) - 2, .025) and -0.0840907710411972 = (467.3003710444578 - 506.5960195534238) / 467.3003710444578 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesMethodBaseHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.145794140086631 > 1.092653023650099. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -42.98397118516 (T) = (0 -1154.454232270694) / Math.Sqrt((104.0476047109114 / (32)) + (35.5393598435794 / (8))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (8) - 2, .025) and -0.11517148190452386 = (1035.2257486885173 - 1154.454232270694) / 1035.2257486885173 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.IsDefinedMethodOverrideHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 516.4559877315426 > 491.35162925303916. IsChangePoint: Marked as a change because one of 5/19/2023 3:32:16 PM, 6/8/2023 8:20:44 PM, 6/29/2023 1:56:51 PM, 7/28/2023 9:24:36 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -15.376493840092877 (T) = (0 -505.07797049901706) / Math.Sqrt((110.28749046002089 / (35)) + (30.51881610300974 / (9))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (9) - 2, .025) and -0.08444277250855625 = (465.74884659949356 - 505.07797049901706) / 465.74884659949356 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.1948879934110415 > 1.131872795878832. IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/9/2023 7:03:36 PM, 7/28/2023 9:24:36 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -31.97201103656371 (T) = (0 -1200.64641080014) / Math.Sqrt((208.07017240165084 / (33)) + (76.30101260911684 / (9))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (9) - 2, .025) and -0.11406351714975492 = (1077.7180944511142 - 1200.64641080014) / 1077.7180944511142 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesClassHitInherit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.2273293870308795 > 1.1792834267881571. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/8/2023 8:02:22 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -23.86113855385404 (T) = (0 -1248.8534093444675) / Math.Sqrt((311.8711372095623 / (36)) + (200.8607217966237 / (8))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (8) - 2, .025) and -0.12490034390710987 = (1110.1902636164482 - 1248.8534093444675) / 1110.1902636164482 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesMethodOverrideHitInherit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.8647952779438135 > 1.8347327861515332. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -27.734351260856663 (T) = (0 -1888.9375074880422) / Math.Sqrt((409.2879488590866 / (34)) + (163.41387745680362 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.09129516652745642 = (1730.913473665163 - 1888.9375074880422) / 1730.913473665163 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.IsDefinedMethodBaseMiss #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 94.87007207979183 > 86.04620853582298. IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -15.470753665636535 (T) = (0 -94.1100538660566) / Math.Sqrt((16.598485164035257 / (36)) + (5.23911958417669 / (8))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (8) - 2, .025) and -0.2101554902192212 = (77.76691063807714 - 94.1100538660566) / 77.76691063807714 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### System.Reflection.Attributes.GetCustomAttributesClassHit #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.126386049351166 > 1048.2970671284065. IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -25.16240065384954 (T) = (0 -1117.9365648787443) / Math.Sqrt((140.5013212658495 / (35)) + (153.08651616783808 / (8))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (8) - 2, .025) and -0.12144814978806084 = (996.8687050667655 - 1117.9365648787443) / 996.8687050667655 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 1da23b146496d95f017439a48f62e78ec15b0289
Compare 60c49c6156e20183e19b87dc7ba8e147674a9f85
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
136.15 ns 159.47 ns 1.17 0.39 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark*'
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark.CreateInstance_0 #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 159.47390086057328 > 148.14123098889073. IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM. IsRegressionStdDev: Marked as regression because -11.199163156427783 (T) = (0 -158.95787242096355) / Math.Sqrt((31.85387088514806 / (34)) + (16.402658535408044 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.13865248927558024 = (139.60174321675072 - 158.95787242096355) / 139.60174321675072 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
cincuranet commented 1 year ago

Look again for System.Reflection.Attributes, maybe PGO needs to catch up with https://github.com/dotnet/runtime/pull/89641.

cincuranet commented 1 year ago

Same on Intel: https://github.com/dotnet/perf-autofiling-issues/issues/20264