Closed andrewlock closed 1 week ago
Branch report: andrew/ci/use-macos-12
Commit report: 6b8bdf1
Test service: dd-trace-dotnet
:white_check_mark: 0 Failed, 342648 Passed, 1643 Skipped, 13h 42m 49.66s Total Time :hourglass: 1 Performance Regression
Baseline
- Samples.HttpMessageHandler.windows.net60.json.scenarios
274.41ms (+8.31ms, +3%) - DetailsExecution-time results for samples comparing the following branches/commits:
Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5707) - mean (73ms) : 65, 81
. : milestone, 73,
master - mean (74ms) : 65, 82
. : milestone, 74,
section CallTarget+Inlining+NGEN
This PR (5707) - mean (988ms) : 966, 1010
. : milestone, 988,
master - mean (987ms) : 965, 1009
. : milestone, 987,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5707) - mean (109ms) : 105, 112
. : milestone, 109,
master - mean (109ms) : 106, 112
. : milestone, 109,
section CallTarget+Inlining+NGEN
This PR (5707) - mean (689ms) : 668, 709
. : milestone, 689,
master - mean (693ms) : 672, 715
. : milestone, 693,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5707) - mean (93ms) : 89, 96
. : milestone, 93,
master - mean (93ms) : 90, 96
. : milestone, 93,
section CallTarget+Inlining+NGEN
This PR (5707) - mean (651ms) : 617, 684
. : milestone, 651,
master - mean (651ms) : 628, 675
. : milestone, 651,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5707) - mean (190ms) : 187, 193
. : milestone, 190,
master - mean (191ms) : 187, 196
. : milestone, 191,
section CallTarget+Inlining+NGEN
This PR (5707) - mean (1,078ms) : 1056, 1101
. : milestone, 1078,
master - mean (1,082ms) : 1060, 1104
. : milestone, 1082,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5707) - mean (276ms) : 272, 280
. : milestone, 276,
master - mean (276ms) : 272, 280
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (5707) - mean (883ms) : 844, 922
. : milestone, 883,
master - mean (865ms) : 840, 890
. : milestone, 865,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5707) - mean (274ms) : 268, 281
. : milestone, 274,
master - mean (265ms) : 261, 269
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (5707) - mean (874ms) : 847, 902
. : milestone, 874,
master - mean (850ms) : 830, 871
. : milestone, 850,
Throughput results for AspNetCoreSimpleController comparing the following branches/commits:
Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.
Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!
gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5707) (11.548M) : 0, 11547601
master (11.466M) : 0, 11465621
benchmarks/2.9.0 (11.627M) : 0, 11626549
section Automatic
This PR (5707) (7.740M) : 0, 7740014
master (7.596M) : 0, 7596011
benchmarks/2.9.0 (8.152M) : 0, 8152375
section Trace stats
master (7.923M) : 0, 7922730
section Manual
This PR (5707) (10.038M) : 0, 10037795
master (9.835M) : 0, 9834507
section Manual + Automatic
This PR (5707) (7.304M) : 0, 7303698
master (7.238M) : 0, 7237913
section Version Conflict
master (6.444M) : 0, 6444217
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5707) (9.440M) : 0, 9439565
master (9.744M) : 0, 9743734
benchmarks/2.9.0 (9.556M) : 0, 9555681
section Automatic
This PR (5707) (6.675M) : 0, 6675024
master (6.723M) : 0, 6722686
section Trace stats
master (6.903M) : 0, 6903372
section Manual
This PR (5707) (8.318M) : 0, 8318129
master (8.254M) : 0, 8253593
section Manual + Automatic
This PR (5707) (6.275M) : 0, 6274555
master (6.249M) : 0, 6249032
section Version Conflict
master (5.649M) : 0, 5648954
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5707) (10.426M) : 0, 10426156
master (10.122M) : 0, 10121981
benchmarks/2.9.0 (10.130M) : 0, 10129519
section Automatic
This PR (5707) (7.244M) : 0, 7244184
master (7.290M) : 0, 7290215
benchmarks/2.9.0 (7.456M) : 0, 7456006
section Trace stats
master (7.537M) : 0, 7537037
section Manual
This PR (5707) (9.027M) : 0, 9026850
master (8.954M) : 0, 8953869
section Manual + Automatic
This PR (5707) (6.981M) : 0, 6981266
master (6.897M) : 0, 6897497
section Version Conflict
master (6.389M) : 0, 6389209
Benchmarks for #5707 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored.
Summary of changes
Change the build agent to use
macos-11
instead ofmacos-12
Reason for change
Microsoft have announced:
And they've started implementing brown-outs, so we need to get off that image ASAP
Implementation details
macos-11
->macos-12
Test coverage
@tonyredondo was going to fire up a macos-11 VM to confirm everything still works when built with the later OS, but he couldn't actually get that to run IIRC - best we can tell it's ok.
Other details
A backport of: