Controlling execution order of unit tests

[shayan0v0n]

Summary

I am suggesting an enhancement to allow developers to set priority or execution order for methods within a single test class. Currently, there is no built-in way to control the execution order of test methods in a class, which can be critical for certain test scenarios.

At present, we have a test class with multiple test methods as shown below:

    [TestClass]
    public class ExampleTests
    {
        [TestMethod]
        public async Task Get_Return_3()
        {
                   //tests
        }

        [TestMethod]
        public async Task Get_Return_1()
        {
                   //tests
        }

        [TestMethod]
        public async Task Get_Return_2()
        {
                   //tests
        }
    }
}

In this scenario, there is no control over the execution order of the test methods. The .NET documentation provides a way to order unit tests using the Priority attribute, but it is somewhat cumbersome and not very intuitive. https://learn.microsoft.com/en-us/dotnet/core/testing/order-unit-tests?pivots=mstest

I propose adding a simple attribute that allows setting the execution priority for each test method within a class. Here is an example of how it might look:

    [TestClass]
    public class ExampleTests
    {
        [TestMethod]
        [Priority(1)]
        public async Task Get_Banner_Return_3()
        {
                   //tests
        }

        [TestMethod]
        [Priority(2)]
        public async Task Get_Banner_Return_1()
        {
                   //tests
        }

        [TestMethod]
        [Priority(3)]
        public async Task Get_Banner_Return_2()
        {
                   //tests
        }
    }
}

[Evangelink]

Hi @shayan0v0n,

I think it makes sense to add this support. We would need to have it under some feature flag to avoid potential breaking change for users.

[shayan0v0n]

I think it makes sense to add this support. We would need to have it under some feature flag to avoid potential breaking change for users.

agree with this

[Evangelink]

I have investigated a little more the current code and the test name order is not even respected.

As for the ordering itself, in MSTest we have 3 behaviors to respect:

• non parallelized tests: defining a customer order here is relatively easy to implement and to grasp functionally.

• parallelized tests at method level: is the ordering working at class level or shall it be considered globally (e.g. all priority methods run before less prioritized methods no matter what's the class they are defined into?)? Shall ordering be disabled here?

• parallelized tests at class level: this is a bit of an hybrid but some questions remains valid.

We had a couple of discussions internally and multiple requests to bring back support for ordered tests and I think this is also falling in the same bucket. Rather than rushing on an implementation, I would prefer to go for a more generic solution that would fit the various use cases.

Moving out of the current iteration so we can work on the design.