andrewlock
commented
3 weeks ago Execution-Time Benchmarks Report :stopwatch:
Execution-time results for samples comparing the following branches/commits:
Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6014) - mean (70ms) : 67, 72
. : milestone, 70,
master - mean (69ms) : 67, 72
. : milestone, 69,
section CallTarget+Inlining+NGEN
This PR (6014) - mean (1,114ms) : 1095, 1133
. : milestone, 1114,
master - mean (1,116ms) : 1094, 1137
. : milestone, 1116,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6014) - mean (108ms) : 105, 111
. : milestone, 108,
master - mean (109ms) : 103, 114
. : milestone, 109,
section CallTarget+Inlining+NGEN
This PR (6014) - mean (814ms) : 801, 827
. : milestone, 814,
master - mean (816ms) : 797, 834
. : milestone, 816,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6014) - mean (93ms) : 90, 95
. : milestone, 93,
master - mean (92ms) : 89, 94
. : milestone, 92,
section CallTarget+Inlining+NGEN
This PR (6014) - mean (770ms) : 755, 785
. : milestone, 770,
master - mean (768ms) : 753, 784
. : milestone, 768,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6014) - mean (190ms) : 188, 193
. : milestone, 190,
master - mean (191ms) : 188, 193
. : milestone, 191,
section CallTarget+Inlining+NGEN
This PR (6014) - mean (1,194ms) : 1171, 1218
. : milestone, 1194,
master - mean (1,194ms) : 1171, 1217
. : milestone, 1194,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6014) - mean (276ms) : 272, 281
. : milestone, 276,
master - mean (276ms) : 271, 280
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (6014) - mean (972ms) : 955, 988
. : milestone, 972,
master - mean (974ms) : 955, 993
. : milestone, 974,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6014) - mean (265ms) : 260, 270
. : milestone, 265,
master - mean (265ms) : 260, 269
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (6014) - mean (955ms) : 927, 982
. : milestone, 955,
master - mean (959ms) : 940, 978
. : milestone, 959,
datadog-ddstaging[bot]
Summary of changes
Fix the diagnostics associated with the expected state of the continuous profiler
Reason for change
In #5240, the continuous profiler gained two new "modes" of operation: delayed enablement based on heuristics, and "monitoring" mode. The tool should be updated to understand those possibilities + handle it in the tests
Implementation details
DD_INJECTION_ENABLEDvariable before running the tests - the presence/absence of the variable changes the expectations, so we need to account for itDD_INJECTION_ENABLEDandDD_PROFILING_ENABLED.Test coverage
Added additional integration and artifact tests
Other details
This should address most of the issues in the scheduled SSI run. I suspect the windows integration tests may need additional work, as currently the profiler library isn't copied to those tests. Will fix in this PR if that ends up being the case
Supersedes