Closed andrewlock closed 2 months ago
Branch report: andrew/ci/fix-installer-tests-2
Commit report: c89b949
Test service: dd-trace-dotnet
:white_check_mark: 0 Failed, 346729 Passed, 1763 Skipped, 14h 1m 24.27s Total Time
Execution-time results for samples comparing the following branches/commits:
Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5994) - mean (74ms) : 64, 83
. : milestone, 74,
master - mean (70ms) : 66, 74
. : milestone, 70,
section CallTarget+Inlining+NGEN
This PR (5994) - mean (1,040ms) : 1018, 1063
. : milestone, 1040,
master - mean (1,084ms) : 1061, 1107
. : milestone, 1084,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5994) - mean (110ms) : 105, 115
. : milestone, 110,
master - mean (109ms) : 105, 112
. : milestone, 109,
section CallTarget+Inlining+NGEN
This PR (5994) - mean (727ms) : 700, 755
. : milestone, 727,
master - mean (763ms) : 740, 786
. : milestone, 763,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5994) - mean (93ms) : 89, 97
. : milestone, 93,
master - mean (93ms) : 89, 97
. : milestone, 93,
section CallTarget+Inlining+NGEN
This PR (5994) - mean (682ms) : 664, 701
. : milestone, 682,
master - mean (714ms) : 694, 733
. : milestone, 714,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5994) - mean (191ms) : 187, 194
. : milestone, 191,
master - mean (191ms) : 188, 195
. : milestone, 191,
section CallTarget+Inlining+NGEN
This PR (5994) - mean (1,118ms) : 1098, 1138
. : milestone, 1118,
master - mean (1,165ms) : 1136, 1193
. : milestone, 1165,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5994) - mean (275ms) : 271, 280
. : milestone, 275,
master - mean (275ms) : 271, 279
. : milestone, 275,
section CallTarget+Inlining+NGEN
This PR (5994) - mean (885ms) : 859, 912
. : milestone, 885,
master - mean (923ms) : 898, 948
. : milestone, 923,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5994) - mean (265ms) : 261, 269
. : milestone, 265,
master - mean (265ms) : 260, 270
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (5994) - mean (869ms) : 841, 897
. : milestone, 869,
master - mean (906ms) : 884, 929
. : milestone, 906,
Benchmarks for #5994 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored.
Summary of changes
Reason for change
The code we were using to install the "local" builds of the NuGet packages added the local source. However the dotnet restore was looking in both the local and nuget.org sources, and installing the highest version it found. That worked fine until we released 3.2.0 publicly and expect to install 2.59.0 of the local build.
For system-tests, we had to change how things work to support v3 (https://github.com/DataDog/system-tests/pull/2962), but unfortunately that breaks for v2.
Implementation details
Install an explicit version of the tracer. As this version is never in the public nuget.org source but is in the local source, it uses that.
For system tests, just pinning to the specific commit prior to the v3 PR being merged.
Having to thread the version through everywhere is kinda horrible, but is the only thing I could find that works.
Test coverage
Running a full installer run here
Other details
The whole approach should be rewritten to be managed by Nuke tbh, if we ever find time
This PR also contains a backport of
Supersedes