search

dotnet / perf-autofiling-issues

A landing place for auto-filed performance issues before they receive triage
MIT License
9 stars 4 forks source link

[Perf] Linux/x64: 30 Regressions on 9/26/2023 11:02:05 PM #22757

Closed performanceautofiler[bot] closed 1 year ago

performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.52 ms 1.82 ms 1.20 0.06 True
2.45 ms 2.69 ms 1.10 0.08 True
1.47 ms 1.72 ms 1.17 0.04 True
3.07 ms 3.70 ms 1.21 0.04 True

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;MyEventsListerViewModel&gt;*'
### Payloads [Baseline]() [Compare]() ### MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.SystemTextJson_Reflection_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.824559012345679 > 1.6116113501417293. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -25.089375529301243 (T) = (0 -1806314.1544189793) / Math.Sqrt((1668704428.930807 / (18)) + (1023455309.6737139 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.18046094113750552 = (1530176.9770361013 - 1806314.1544189793) / 1530176.9770361013 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.JsonNet_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 2.6927960277777783 > 2.493151052088236. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -22.90502193214126 (T) = (0 -2765261.413756244) / Math.Sqrt((4990068025.959999 / (18)) + (3201041372.523727 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.18761425307748728 = (2328417.1662562736 - 2765261.413756244) / 2328417.1662562736 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.SystemTextJson_SourceGen_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.7194115666666667 > 1.5642453402982959. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -28.79844243838583 (T) = (0 -1704578.6549522462) / Math.Sqrt((875517244.8300786 / (18)) + (558681860.0398955 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.1559124238547229 = (1474660.7266905548 - 1704578.6549522462) / 1474660.7266905548 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.DataContractJsonSerializer_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.700206057575757 > 3.217898566744599. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -36.23297484071996 (T) = (0 -3727027.159834185) / Math.Sqrt((5240209639.663979 / (18)) + (3651426183.4413457 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.23753349471981022 = (3011657.6042073276 - 3727027.159834185) / 3011657.6042073276 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.TimeToFirstService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
24.95 ฮผs 26.86 ฮผs 1.08 0.06 True
22.97 ฮผs 25.56 ฮผs 1.11 0.05 True
22.69 ฮผs 25.99 ฮผs 1.15 0.06 True
23.38 ฮผs 25.41 ฮผs 1.09 0.06 True
5.68 ฮผs 6.60 ฮผs 1.16 0.12 True
23.25 ฮผs 26.31 ฮผs 1.13 0.06 True
23.17 ฮผs 25.38 ฮผs 1.10 0.07 True
5.54 ฮผs 6.58 ฮผs 1.19 0.10 True
5.88 ฮผs 6.63 ฮผs 1.13 0.09 True
22.59 ฮผs 25.75 ฮผs 1.14 0.06 True
24.83 ฮผs 27.87 ฮผs 1.12 0.06 True
23.15 ฮผs 26.77 ฮผs 1.16 0.06 True
22.85 ฮผs 26.20 ฮผs 1.15 0.06 True
24.55 ฮผs 27.74 ฮผs 1.13 0.02 True
5.66 ฮผs 6.83 ฮผs 1.21 0.07 True

graph graph graph graph graph graph graph graph graph graph graph graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.TimeToFirstService*'
### Payloads [Baseline]() [Compare]() ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Dynamic") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 26.85971734796328 > 26.166876279524008. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -20.157827334495664 (T) = (0 -27686.4792240054) / Math.Sqrt((352753.4315570619 / (18)) + (184762.73220685558 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.12929876025499462 = (24516.523172091165 - 27686.4792240054) / 24516.523172091165 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Expressions") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 25.555841556836903 > 24.321916697289797. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -16.869874477315424 (T) = (0 -25878.1610729127) / Math.Sqrt((487244.8680121465 / (18)) + (216862.25637765074 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.134942066860962 = (22801.305748131148 - 25878.1610729127) / 22801.305748131148 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Runtime") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 25.991583379710743 > 23.58149417316256. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -25.759332249660023 (T) = (0 -26104.26708154246) / Math.Sqrt((244048.3299168993 / (17)) + (174374.36340277136 / (37))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (37) - 2, .025) and -0.15776176561162178 = (22547.183588976193 - 26104.26708154246) / 22547.183588976193 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Dynamic") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 25.405235403138526 > 24.56202146198394. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -15.528883439560005 (T) = (0 -25851.51281972971) / Math.Sqrt((515007.9581119897 / (18)) + (223252.17838539707 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.1258840471532995 = (22961.079238215538 - 25851.51281972971) / 22961.079238215538 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.BuildProvider(Mode: "Dynamic") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.601703882092199 > 5.9428599805803834. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -22.178716634551748 (T) = (0 -6642.509610163102) / Math.Sqrt((20664.377243237395 / (18)) + (30240.568194753803 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.17491302107519646 = (5653.61817514317 - 6642.509610163102) / 5653.61817514317 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Runtime") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 26.306871170957766 > 24.563144572981. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -14.895044292200968 (T) = (0 -25738.606155213987) / Math.Sqrt((631065.7939863703 / (18)) + (208068.02615781318 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.13214941673373612 = (22734.283809879227 - 25738.606155213987) / 22734.283809879227 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "ILEmit") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 25.377423559423768 > 24.33436939586148. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -18.463353404773713 (T) = (0 -25610.772941042585) / Math.Sqrt((342616.2974370791 / (17)) + (251141.72727855493 / (37))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (37) - 2, .025) and -0.1342126260104308 = (22580.222044545513 - 25610.772941042585) / 22580.222044545513 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.BuildProvider(Mode: "ILEmit") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.583962315284015 > 5.959219058953828. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -28.467756090919256 (T) = (0 -6677.023866346851) / Math.Sqrt((13772.560353775603 / (17)) + (25309.875422435958 / (37))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (37) - 2, .025) and -0.1973276119034718 = (5576.605600644204 - 6677.023866346851) / 5576.605600644204 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.BuildProvider(Mode: "Runtime") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.631136256121155 > 5.964462933698679. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -27.10802326542417 (T) = (0 -6658.517389920106) / Math.Sqrt((18506.868434289718 / (18)) + (15412.612870413872 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.18307243486176417 = (5628.1569865991505 - 6658.517389920106) / 5628.1569865991505 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Dynamic") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 25.74825047648146 > 23.8279640005814. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -20.597179388617196 (T) = (0 -25511.138976878858) / Math.Sqrt((258520.97652536054 / (18)) + (208479.00268885263 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.12888699667093043 = (22598.48776016625 - 25511.138976878858) / 22598.48776016625 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Runtime") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.866563135514877 > 26.0229101704959. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -19.24152182628083 (T) = (0 -27930.02176162699) / Math.Sqrt((406945.3780758713 / (18)) + (329143.65417442715 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.13931350228808076 = (24514.781669430926 - 27930.02176162699) / 24514.781669430926 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "ILEmit") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 26.768301359953703 > 24.671220819777695. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -19.616078253243245 (T) = (0 -26202.36345109094) / Math.Sqrt((408115.2645538244 / (18)) + (217305.11166727048 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.14480259106471374 = (22888.106347419827 - 26202.36345109094) / 22888.106347419827 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Expressions") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 26.198342954029016 > 23.93056242539411. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -18.441222267533064 (T) = (0 -25443.732425340557) / Math.Sqrt((312431.9958452222 / (18)) + (277617.8075119539 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.12902287082818256 = (22536.0646650821 - 25443.732425340557) / 22536.0646650821 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Expressions") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 27.73881370626215 > 26.191787570665518. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -19.664299154636183 (T) = (0 -27961.26472034789) / Math.Sqrt((455112.8308082702 / (18)) + (192263.9653503924 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.1399664124288631 = (24528.147860753525 - 27961.26472034789) / 24528.147860753525 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Microsoft.Extensions.DependencyInjection.TimeToFirstService.BuildProvider(Mode: "Expressions") #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 6.834767578970142 > 5.961402423646312. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -25.523485361356755 (T) = (0 -6560.410544101565) / Math.Sqrt((14768.65975806586 / (18)) + (19461.991282767474 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.1665355842137121 = (5623.840912254317 - 6560.410544101565) / 5623.840912254317 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
performanceautofiler[bot] commented 1 year ago

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.Sort<BigStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
548.40 ฮผs 851.70 ฮผs 1.55 0.11 True
1.46 ms 1.87 ms 1.28 0.04 True
1.44 ms 1.92 ms 1.33 0.04 True

graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;BigStruct&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.Sort<BigStruct>.Array_Comparison(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 851.6956333166667 > 578.4593284373078. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -52.76552481119796 (T) = (0 -868597.8572696632) / Math.Sqrt((403049097.2046477 / (18)) + (507007170.9980325 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.5794920667615887 = (549922.2664983292 - 868597.8572696632) / 549922.2664983292 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Sort<BigStruct>.LinqOrderByExtension(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.8729003713999999 > 1.5190412789720003. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -39.276332503334544 (T) = (0 -1913894.1165468805) / Math.Sqrt((526514714.32992125 / (18)) + (4204318601.376885 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.3347810405213801 = (1433863.7262927352 - 1913894.1165468805) / 1433863.7262927352 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.Sort<BigStruct>.LinqQuery(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.9198149572 > 1.523386837028. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -56.821982275905185 (T) = (0 -1910779.1975268787) / Math.Sqrt((231019849.37224564 / (18)) + (2066991052.008134 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.3271043734030021 = (1439810.7909381683 - 1910779.1975268787) / 1439810.7909381683 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.13 ฮผs 4.45 ฮผs 1.08 0.06 True
2.97 ฮผs 3.35 ฮผs 1.13 0.10 True
3.36 ฮผs 3.62 ฮผs 1.08 0.12 True
2.70 ฮผs 3.02 ฮผs 1.12 0.12 False

graph graph graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>.DeserializeFromReader(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 4.449273076063094 > 4.281559023538991. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -16.935469181876392 (T) = (0 -4462.464485939356) / Math.Sqrt((9607.220716981787 / (18)) + (6984.836501461708 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.11454063227544184 = (4003.859847468105 - 4462.464485939356) / 4003.859847468105 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>.DeserializeFromUtf8Bytes(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.346678903101612 > 3.110892678640051. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -12.95610595578692 (T) = (0 -3292.9599634716524) / Math.Sqrt((17530.021300008062 / (17)) + (4554.411049998137 / (37))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (37) - 2, .025) and -0.15429774364264986 = (2852.7821193516033 - 3292.9599634716524) / 2852.7821193516033 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>.DeserializeFromString(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.6249621398407013 > 3.616398227334998. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -12.456554472812115 (T) = (0 -3758.213089202795) / Math.Sqrt((19966.887635381336 / (18)) + (5817.06345570014 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.13372934228557554 = (3314.9120773625855 - 3758.213089202795) / 3314.9120773625855 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>.DeserializeFromUtf8Bytes(Mode: SourceGen) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 3.020753184703062 > 2.9051907237385923. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -11.36767949784177 (T) = (0 -2957.701287069674) / Math.Sqrt((6436.513125473647 / (18)) + (3363.861518533062 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.0886073231987266 = (2716.958837259027 - 2957.701287069674) / 2716.958837259027 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.10 ฮผs 1.20 ฮผs 1.09 0.11 False

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeObjectProperty(Mode: Reflection) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 1.2037353228680956 > 1.1732060889359928. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -11.686582485258322 (T) = (0 -1208.5330782883264) / Math.Sqrt((1538.5984516098135 / (18)) + (965.8583210489072 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.11461297185699984 = (1084.2625277138584 - 1208.5330782883264) / 1084.2625277138584 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
170.83 ฮผs 238.08 ฮผs 1.39 0.09 True
421.19 ฮผs 562.44 ฮผs 1.34 0.18 True

graph graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'
### Payloads [Baseline]() [Compare]() ### System.Collections.TryGetValueTrue<Int32, Int32>.SortedList(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 238.07612208624707 > 178.45863792009993. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -63.13938677640466 (T) = (0 -240165.8933380952) / Math.Sqrt((12017589.076049907 / (18)) + (23565733.5352994 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.4332911689186623 = (167562.5291958555 - 240165.8933380952) / 167562.5291958555 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### System.Collections.TryGetValueTrue<Int32, Int32>.SortedDictionary(Size: 512) #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 562.4400507015306 > 445.58034917300114. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -36.296940482062574 (T) = (0 -589636.2677649085) / Math.Sqrt((252826396.8924974 / (18)) + (247009755.9676643 / (37))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (37) - 2, .025) and -0.38931460670703716 = (424408.0245888067 - 589636.2677649085) / 424408.0245888067 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Compare da5bd2bc91c78a457c4bf28c09759f368e0f3238
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.96 ฮผs 12.55 ฮผs 1.15 0.09 True

graph Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;Location&gt;*'
### Payloads [Baseline]() [Compare]() ### MicroBenchmarks.Serializers.Xml_ToStream<Location>.XmlSerializer_ #### ETL Files #### Histogram #### Description of detection logic ``` IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small. IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline. IsRegressionWindowed: Marked as regression because 12.551794405664447 > 11.776484056606515. IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 10/4/2023 10:43:01 PM falls between 9/26/2023 10:37:55 AM and 10/4/2023 10:43:01 PM. IsRegressionStdDev: Marked as regression because -18.163907928688108 (T) = (0 -12659.582652183359) / Math.Sqrt((131760.7251324571 / (18)) + (45470.17202769107 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.15330494322431423 = (10976.786951758611 - 12659.582652183359) / 10976.786951758611 is less than -0.05. IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small. IsChangeEdgeDetector: Marked as regression because Edge Detector said so. ``` #### JIT Disasms ### Docs [Profiling workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/profiling-workflow-dotnet-runtime.md) [Benchmarking workflow for dotnet/runtime repository](https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-dotnet-runtime.md)
radekdoulik commented 1 year ago

duplicate of https://github.com/dotnet/perf-autofiling-issues/issues/22385