microsoft / vstest

Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
MIT License
869 stars 316 forks source link

The InProcDataCollector's TestCaseStart/TestCaseStop methods are not always called. #4997

Open kname88 opened 2 months ago

kname88 commented 2 months ago

Description

I've implemented example InProcDataCollector in order to track the test-case execution but its methods: TestCaseStart/TestCaseStop are not called for some data-driven tests. Below is a scenario where 8 tests are executed but the InProcDataCollector is 'informed' only about 6 of them.

Steps to reproduce

Expected behavior

Expecting 8 tests to be executed and SimpleDataCollector's TestCaseStart/TestCaseStop methods are triggered for each one, so 8 times. Traces expected to be reported by SimpleDataCollector are:

    TestSessionStart : <Configuration><Port>4312</Port></Configuration>
    TestCaseStart : TestRowsWithString_1
    TestCaseEnd : TestRowsWithString_1
    TestCaseStart : TestRowsWithString_2
    TestCaseEnd : TestRowsWithString_2
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseStart : TestRowsWithMixData_DataRow1
    TestCaseEnd : TestRowsWithMixData_DataRow1
    TestCaseStart : TestRowsWithMixData_DataRow2
    TestCaseEnd : TestRowsWithMixData_DataRow2
    TestCaseStart : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow11) parameters
    TestCaseEnd : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow11) parameters
    TestCaseStart : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow12) parameters
    TestSessionEnd

Actual behavior

Executed 8 tests but SimpleDataCollector's TestCaseStart/TestCaseStop methods are triggered only 6 times:

   TestSessionStart : <Configuration><Port>4312</Port></Configuration>
    TestCaseStart : TestRowsWithString_1
    TestCaseEnd : TestRowsWithString_1
    TestCaseStart : TestRowsWithString_2
    TestCaseEnd : TestRowsWithString_2
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseStart : TestRowsWithMixData
    TestCaseEnd : TestRowsWithMixData
    TestCaseStart : TestWithDynamicMixDataArgs
    TestCaseEnd : TestWithDynamicMixDataArgs
    TestSessionEnd

InProcDataCollector-CasePicture1

Environment

Windows10 Enterprise
VS2017.8 and .Net8 SDK
NuGet pacakges (see the attached projects):
    Microsoft.TestPlatform.XXX v17.8.0
    MSTest.XXX v3.3.1