Add support for ordered tests

[sbaid]

Description

Is it possible to decide this based on telemetry data of ordered tests usage?

Steps to reproduce

What steps can reproduce the defect? Please share the setup, sample project, target platform (desktop, core, uap)

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment. Operating system, Build version of vstest.console, Package version of MSTest framework and adapter

[sbaid]

@harshjain2 Do we have telemetry data of usage of ordered tests, generic tests?

[sbaid]

@pvlakshm to keep u in loop

[AbhitejJohn]

I believe @pvlakshm has also filed a user voice item for this.

[codito]

https://visualstudio.uservoice.com/forums/121579-visual-studio-ide/suggestions/16417609-enable-ordered-test-support-for-mstest-v2-framewor

[pvlakshm]

We should support ordered tests in a generic manner and not restrict it to only MSTest framework based tests - perhaps as yet another extension point offered by the test platform.

[divijkumar]

We have an app (on multiple platforms) where a lot of the code is in C++. We use a very thin cross-platform wrapper for writing unit tests. The wrapper internally uses CPPUnitTestFramework for unit testing on Windows.

Sometimes, when running the unit tests one of the tests will fail, and that will result in some of the tests that are executed after the failed test, to fail as well. Most of the times the fix is a trivial one, but time is spent in segregating the test which actually failed from those that failed as a result of some remnant of the earlier failed test.

We would like to know if there is a way to determine the order in which the unit tests are run inside Visual Studio. This will help us in determining the failing test quickly, and help resolve issues faster.

Moreover, it would be good to have a mechanism to control the order of the execution of the unit tests inside Visual Studio.

[AbhitejJohn]

@divijkumar : Ideally tests should be isolated units and should not adversely affect other tests. They do so by ensuring that the right cleanup is done even if a test fails. When this is not possible for some reason, you could turn on diagnostic logs(link for the default test platform, link for the new cross plat test platform) to figure out what order they actually ran. Yes, this issue should probably cover providing ordered test support for C++ based tests as well.

[rhundhausen]

I'm not a huge fan of ordered tests, but they do serve a purpose (e.g. integration testing by sequencing many unit tests). That said, one would expect the default behavior of VS 2017 to just work.

[JDCain]

I wanted to know if this was any plan for adding built in support for testorder to the roadmap. We currently have some workarounds but they are rather ugly implementations with custom scripts reading a defined order and calling vstest.console.exe for each individual test.

[melleck]

Come on guys, it has been over 2 years now and there is still no update/fix0 on this issue? Ordered tests are now a norm especially for Selenium based GUI tests developed using VS C#. I had to ditch MSTest in favour of NUnit just because MSTest didn't support ordered tests.

[xdevs23]

What do you expect from Microsoft, they don't really care about this. I do need this feature but looks like I need to solve it differently.... I hate working with C#... and Microsoft products sigh

[PBoraMSFT]

@JDCain - it's on the radar (which is why the issue remains open), but it's unlikely that we will get to it in CY19 Q2. We are open to taking contributions if anyone is interested.

[dondrakon]

Any update of this task?

[JDCain]

Any update of this task?

I ended up going to Xunit and writing a test dependency orderer. https://github.com/JDCain/Xunit.Extensions.TestDependency

[saiparth]

Any update on this?

[N-Olbert]

While not equivalent to ordered tests it seems that at least an alphanumerical execution order is defined behaviour according to docs and samples

[nalamTekhqs]

any update regarding this issue?

[deepasajin]

Any update on this ticket. I beleive we have an option to order the test using XML format. Any one tried that ?

[Haplois]

@deepasajin, Ordered tests are not currently supported by MSTestV2; based on telemetry, it's not currently on our roadmap. Also, it is the best practice to avoid ordering tests.

[nalamTekhqs]

Thanks @Haplois for the update

[saiparth]

Hi @Haplois Any plan to support external files?(if testnames are kept in external file) I may have 1000 tests which needs to be run in those order. I cant run parallelly or without order because application will not support that. Ex: dotnet test --filter tests.csv if not is there any limit for mentioning testname in this?if i create a file with 1000 test name, will that work? Ex dotnet test --filter tests1,test2.....

[Haplois]

@saiparth, providing a filter doesn't guarantee that tests will run in given order -- it is just for filtering which tests to run. There is a maximum limit set by the OS when using --filter (see: https://github.com/microsoft/vstest/issues/2764). We have an unsupported and undocumented way of filtering tests using the .runsettings file. You can try using that with the understanding of it might break in the future. (see https://github.com/microsoft/vstest/pull/2356#issuecomment-603803917)

[N-Olbert]

@Evangelink Due to the conversation in #821 maybe this issue should be reopened

[Evangelink]

Reopened for investigation if that's something we want to support.

[WhitWaldo]

I'd like to see support for ordered tests primarily to ensure that I can execute a set of smaller unit tests sequentially comprising a larger integration test.