Closed andrewlock closed 3 months ago
datadog-ddstaging[bot]
commented
3 months ago Branch report: andrew/ci/masstransit-fix-backport
Commit report: ce7f3a8
Test service: dd-trace-dotnet
:white_check_mark: 0 Failed, 353745 Passed, 1797 Skipped, 14h 35m 6.14s Total Time
andrewlock
commented
3 months ago Execution-time results for samples comparing the following branches/commits:
Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5911) - mean (77ms) : 62, 92
. : milestone, 77,
section CallTarget+Inlining+NGEN
This PR (5911) - mean (1,072ms) : 1050, 1093
. : milestone, 1072,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5911) - mean (111ms) : 107, 114
. : milestone, 111,
section CallTarget+Inlining+NGEN
This PR (5911) - mean (782ms) : 763, 801
. : milestone, 782,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5911) - mean (94ms) : 92, 97
. : milestone, 94,
section CallTarget+Inlining+NGEN
This PR (5911) - mean (723ms) : 705, 742
. : milestone, 723,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5911) - mean (191ms) : 188, 195
. : milestone, 191,
section CallTarget+Inlining+NGEN
This PR (5911) - mean (1,155ms) : 1128, 1182
. : milestone, 1155,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5911) - mean (277ms) : 272, 281
. : milestone, 277,
section CallTarget+Inlining+NGEN
This PR (5911) - mean (947ms) : 926, 969
. : milestone, 947,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5911) - mean (265ms) : 260, 270
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (5911) - mean (930ms) : 908, 951
. : milestone, 930,
Benchmarks for #5911 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored.
Summary of changes
Skip the mass transit smoke test as it seems to be a cause of a lot of flakiness
Reason for change
We've seen a lot of errors in the
CheckBuildlogsForErrstage:These seemed to get a lot worse after we disabled keep-alive, but that's anecdotal.
Implementation details
It's not entirely clear if the problem is just coincidentally related to the MassTransit test (i.e. it's a test ordering process) or if it's actually something about the test.
As a check I tried skipping the test in this branch and did 4 full (all TFM) integration tests runs, and didn't see the issue again. It's all still anecdotal, but rather trade off flakiness here. If the problem reappears subsequently, we can look into it again further.
Test coverage
Did 4 full runs, and didn't see the issue again
Other details
Backport of #5861 (as still getting a lot of flake on the release/2.x branch)