Make sure we run all the TFMs on master builds

andrewlock commented 3 weeks ago

Summary of changes

Update RequiresThoroughTesting() to return true when we're running on master or release/2.x

Reason for change

We should be doing thorough testing on the main branches. I thought we were. I don't quite understand how we're not

Implementation details

Check to see if we're on the main branch for the verison, and if so, do the thorough testing

Test coverage

Can't really test this properly until we merge it...

Other details

I sure hope this doesn't reveal more issues :awkward:

andrewlock commented 3 weeks ago

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5990) - mean (71ms)  : 67, 74
     .   : milestone, 71,
    master - mean (69ms)  : 67, 71
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (5990) - mean (1,088ms)  : 1058, 1119
     .   : milestone, 1088,
    master - mean (1,077ms)  : 1053, 1101
     .   : milestone, 1077,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5990) - mean (109ms)  : 105, 114
     .   : milestone, 109,
    master - mean (109ms)  : 105, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5990) - mean (761ms)  : 741, 781
     .   : milestone, 761,
    master - mean (758ms)  : 739, 778
     .   : milestone, 758,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5990) - mean (93ms)  : 89, 97
     .   : milestone, 93,
    master - mean (92ms)  : 90, 95
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (5990) - mean (715ms)  : 691, 739
     .   : milestone, 715,
    master - mean (710ms)  : 697, 724
     .   : milestone, 710,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5990) - mean (191ms)  : 187, 195
     .   : milestone, 191,
    master - mean (190ms)  : 187, 192
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (5990) - mean (1,161ms)  : 1137, 1185
     .   : milestone, 1161,
    master - mean (1,160ms)  : 1137, 1184
     .   : milestone, 1160,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5990) - mean (274ms)  : 270, 279
     .   : milestone, 274,
    master - mean (274ms)  : 270, 279
     .   : milestone, 274,

    section CallTarget+Inlining+NGEN
    This PR (5990) - mean (917ms)  : 899, 935
     .   : milestone, 917,
    master - mean (922ms)  : 902, 942
     .   : milestone, 922,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5990) - mean (263ms)  : 259, 267
     .   : milestone, 263,
    master - mean (264ms)  : 259, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (5990) - mean (905ms)  : 884, 926
     .   : milestone, 905,
    master - mean (902ms)  : 883, 922
     .   : milestone, 902,

datadog-ddstaging[bot] commented 3 weeks ago

Datadog Report

Branch report: andrew/ensure-we-run-all-tfms-in-integration-tests-master Commit report: e3a3368 Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 366271 Passed, 2059 Skipped, 14h 53m 53.3s Total Time

DataDog / dd-trace-dotnet