Test Adapter: parametrized tests are discovered/logged to data collector as 1 test (no information about specific parameters)

pkrukp commented 6 years ago

Description

Hi,

MsTest tests may be parametrizable, for example a method with DataTestMethod and DataRow attributes. For example:

        [DataTestMethod]
        [DataRow("baz", false)]
        [DataRow("foo", true)]
        public void TestMethod(string name, bool expectedResult)

Current behavior

For such test, MsTest Test Adapter:

during test discovery, only 1 VSTest TestCase is discovered (which represents whole TestMethod)
during test execution, the method is executed for data row, but VSTest's DataCollectors (both In-Proc and out of process) receive only 1 TestCaseStart/TestCaseStop event (for TestMethod)

Desired behavior

Would it be possible to have per data row discovery/logging resolution? That is, ideally:

discovery would discover 2 test cases: TestMethod("baz", false) and TestMethod("foo", true)
DataCollector would receive TestCaseStart/TestCaseStop for both above test cases

In fact, the 2nd point is more important for me - I would like to change environment and monitor each execution of TestMethod (for each data row separately).

If it is not possible, could you please share the reason?

Thanks

Environment

mstest.testadapter.1.2.0 microsoft.testplatform.15.6.0-preview-20171211-02

cltshivash commented 6 years ago

@pkruk2 Having separate tests for each datarow is not in plan for the near future. Would it be possible to check the parameter value in your test code (or) does this need to be done outside the test code ?

pkrukp commented 6 years ago

@cltshivash It must be outside of the test code.

In case I need to add this myself to the test adapter, do you have any suggestions? Ideally it'd be accepted upstream.

I understand it'd also change VS behavior, because currently when you discover/run such parametrized test, VS shows 1 test in Test Explorer, and all data rows results are visible in test details section.

jayaranigarg commented 6 years ago

Tagging @pvlakshm , @cltshivash to see if and how can we bring this in?

AbhitejJohn commented 6 years ago

@jayaranigarg : Another reason why it would be really nice to have separate test cases is in #171. To be able to run/debug an individual test case would greatly improve dev efficiency especially if these are long running tests.

ShreyasRmsft commented 6 years ago

@pkruk2 this is by design and it is a non-goal to have discovery by data row.

But your ask for DataCollectors to be aware of iterations is a valid ask and can be solved (Given the constraint of not discovering each data row as a separate test method) is for us to add new events for test iteration start and test iteration end for data collectors to hook in to. But this will require a breaking change in the test Platform object model.

A simple solution is for us to have custom events in the Test Platform for datacollectors. Anyone who wants to send custom events from their adapters (in our case MsTestv2) can use this Extensible List of events to register and listen to custom events like test iteration start and test iteration end. But this won't be a simple thing to implement.

jayaranigarg commented 5 years ago

Closing this as duplicate. Having seperate test method for each data row will fix this As mentioned in #1 of this issue : https://github.com/Microsoft/testfx/issues/464

microsoft / testfx