How to triger the MSTestV2 programatically?

[saiparth]

I am using specflow with MsTestV1. I am using orderedTest to run. Since V2 does not support orderedTests, is there any way to Triger the tests programmatically in V2. I checked the unit tests, I dint get it I am trying like this https://social.msdn.microsoft.com/Forums/vstudio/en-US/ccd08ce1-a86b-4965-af68-1db8cd7d6715/how-to-execute-test-methods-programmatically-using-c-may-be-using-reflection?forum=vsunittest

For V2 I refered this. but not getting it https://github.com/microsoft/testfx/blob/master/test/UnitTests/MSTest.CoreAdapter.Unit.Tests/MSTestExecutorTests.cs

I dont want to run using command line..the test name will be in a list,I have to invoke them.

[nohwnd]

Have you tried the vstest console wrapper? Or is that something you want to avoid as well? https://github.com/nohwnd/vstest-examples/blob/main/PartioningTests/Program.cs#L73 in mstestv2 tests are ordered based on method names btw.

[saiparth]

i tried like this - https://github.com/saiparth/consoleRunner but its not discovering tests. through console its working using dotnet test command

[saiparth]

@nohwnd Thank you. I was able to fix and run like shown in example. please suggest for below scenario. I have a framework which is build in C# (Version=v4.6.1) this uses specflow and mstest V1. It uses orderedtest for grouping test. already hundreds of tests are there so I cant avoid ordertests. problem is mstest V2 will not support orderedtests. so I am thinking to use like vstest.console.exe sample.orderedtest /logger:trx to run. is this running ordertests supported in future or is there any other option to keep the tests in external file and use that. filtering will not work . I have checked. Please suggest.

[N-Olbert]

@saiparth Support for ordered tests in the future seems to be unlikely (see https://github.com/microsoft/testfx/issues/25#issuecomment-790955113). We fixed our (few) tests which really require ordering by merging them into a single test method which executes them sequentially.

[saiparth]

@N-Olbert My problem is that I am using specflow which behind the screen generates the testmethods. and same test case can be present in Smoke test, regression multiple suites. so tags/ alphabetical order will not work. only way is I have to put tests in external file and invoke them programmatically. using Nunit its possible to do like this.

ITestEngine engine = TestEngineActivator.CreateInstance();
TestPackage package = new TestPackage("TestProjectdllPath");
ITestRunner runner = engine.GetRunner(package);
TestFilterBuilder fil = new TestFilterBuilder();
fil.AddTest("NameSpace.TestName");
TestFilter filter = fil.GetFilter();
XmlNode result = runner.Run(new MyEventListener(), filter);
runner.Dispose();

since my framework was using Mstest V1, I thought of continuing MstestV2

[Evangelink]

@saiparth Is it something you are still interested in having?

[saiparth]

Yes, Is there solution?

[Evangelink]

Not sure, I need to look into it. I just wanted to confirm there was a need for it.

Could you confirm the kind of usage you would like to achieve? Because depending on them, there might be a need to use Test Platform (or not).

[saiparth]

I am looking for support of orderedtest in MSTEST V2. There was an option in V1 which displayed all unit test in project and we could generate a xml file by picking specific testcase. Good thing was execution order was same as test names in external file even though it was the list was not alphabetical order.

Using some tools we can run the test from external files. But it wont execute in the order that we mention. Its not possible to mention the execution order for the test using attributes due to various reasons. The tests would execute randomly. Some of the test wont support parallel execution and we need sequential execution. Also the test list may not be in alphabetical order

[Evangelink]

I will make tests but I fear that to run in order you would need to call MSTest n times with 1 test per call.

[N-Olbert]

@Evangelink Since you were interested in this issue and seem make a roadmap for MSTest v3 and v4.... are there any plans to fix the main reason for this request and provide a way to specify the execution order of the tests (by an attribute, a custom IComparer-instance that has to be registered somewhere or whatever). Even if the corresponding issue #25 has already been closed, I think this is an important feature for integration and acceptance testing. In fact this lack of functionality is the main reason i would struggle to use MSTest in a new project.

[Evangelink]

Hey @N-Olbert. Yes, I'd like to work on a feature to give users the ability to have better control on the order of their tests but I haven't yet given enough thoughts to suggest something now. For sure, it won't be part of v3, I will see if it could fit v4.

[Evangelink]

I had a quick look and sadly with current architecture, there is no way for you to control the order. As I said previously, you would need to make a lot of calls with 1 assembly and a filter to match only 1 test of that assembly.

As for the programmatically doing the calls, you can:

1. Make multiple calls to vstest.console.exe or dotnet test with 1 source and a filter for each test. See https://learn.microsoft.com/visualstudio/test/vstest-console-options and https://learn.microsoft.com/dotnet/core/tools/dotnet-test.

2. Reference Microsoft.TestPlatform.TranslationLayer NuGet package and call the available APIs. You can use our E2E tests as example (see https://github.com/microsoft/testfx/blob/main/test/E2ETests/Automation.CLI/CLITestBase.e2e.cs#L51).

3. Call directly MSTestExecutor class from the test adapter dll. This last option might be easiest but you will lose all the extra features (blame, trx, logs...) provided by the test platform.

Let me know if you need more information.

[Evangelink]

Hey there, as you may have noticed we are about to start shipping MSTest runner and this runner will greatly facilitate most of the requests open in this issue.

Please look at the code and let us know what you think.