Closed andrewlock closed 3 weeks ago
Execution-time results for samples comparing the following branches/commits:
Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5988) - mean (70ms) : 67, 73
. : milestone, 70,
master - mean (69ms) : 67, 71
. : milestone, 69,
section CallTarget+Inlining+NGEN
This PR (5988) - mean (1,080ms) : 1052, 1107
. : milestone, 1080,
master - mean (1,077ms) : 1053, 1101
. : milestone, 1077,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5988) - mean (109ms) : 105, 113
. : milestone, 109,
master - mean (109ms) : 105, 112
. : milestone, 109,
section CallTarget+Inlining+NGEN
This PR (5988) - mean (759ms) : 730, 787
. : milestone, 759,
master - mean (758ms) : 739, 778
. : milestone, 758,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5988) - mean (92ms) : 89, 95
. : milestone, 92,
master - mean (92ms) : 90, 95
. : milestone, 92,
section CallTarget+Inlining+NGEN
This PR (5988) - mean (710ms) : 690, 729
. : milestone, 710,
master - mean (710ms) : 697, 724
. : milestone, 710,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5988) - mean (190ms) : 187, 194
. : milestone, 190,
master - mean (190ms) : 187, 192
. : milestone, 190,
section CallTarget+Inlining+NGEN
This PR (5988) - mean (1,158ms) : 1128, 1187
. : milestone, 1158,
master - mean (1,160ms) : 1137, 1184
. : milestone, 1160,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5988) - mean (276ms) : 271, 280
. : milestone, 276,
master - mean (274ms) : 270, 279
. : milestone, 274,
section CallTarget+Inlining+NGEN
This PR (5988) - mean (922ms) : 902, 942
. : milestone, 922,
master - mean (922ms) : 902, 942
. : milestone, 922,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5988) - mean (265ms) : 260, 269
. : milestone, 265,
master - mean (264ms) : 259, 268
. : milestone, 264,
section CallTarget+Inlining+NGEN
This PR (5988) - mean (903ms) : 883, 923
. : milestone, 903,
master - mean (902ms) : 883, 922
. : milestone, 902,
Branch report: andrew/ci/improve-snapshot-output
Commit report: 7bffafb
Test service: dd-trace-dotnet
:white_check_mark: 0 Failed, 363637 Passed, 2072 Skipped, 14h 50m 59.88s Total Time
Benchmarks for #5988 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored.
Summary of changes
Improve the output when installer snapshot tests fail
Reason for change
Currently we just show a fixed string, this means we show an actual diff instead
Implementation details
We already do this for the macos installer tests, this improves it, so that you get something like this:
Test coverage
It won't fail in this test (hopefully) but we already know this works elsewhere so meh