Open albert-yai opened 1 month ago
Hi @albert-yai,
Could you please confirm that running the test locally either by dotnet test
or vstest.console.exe
on the project produces the following output:
29/05/2024 09:45:07 : AssemblyInitialize
29/05/2024 09:45:08 : TestInitialize
29/05/2024 09:45:09 : TestMethod
29/05/2024 09:45:10 : TestCleanup
29/05/2024 09:45:11 : AssemblyCleanup
I'll investigate the behavior on AzDO to understand how VSTest calls us.
@Evangelink
Yes, running the test locally with vstest.console.exe produces the following output:
5/29/2024 11:53:40 AM : AssemblyInitialize
5/29/2024 11:53:45 AM : TestInitialize
5/29/2024 11:53:50 AM : TestMethod
5/29/2024 11:53:55 AM : TestCleanup
5/29/2024 11:54:00 AM : AssemblyCleanup
@engyebrahim could you please investigate this issue?
Describe the bug
MSTest AssemblyCleanup executes immediately after the first test point in multi-test-point test run via Azure Devops Release pipeline VsTest Task. The bug only occurs when all tests in the test run are associated to the same MSTest TestMethod.
Version Info bug is present on MSTest.TestFramework version >= 3.3.1 bug is not present on MSTest.TestFramework v2.2.3
Steps To Reproduce
Outcome: AssemblyCleanup executes immediately after the first test point completes.
Minimum reproducible example Visual Studio 2019 MSTest Project MSTestProject.zip
Example MSTest implementation
Using the above example, execute Azure DevOps Release pipeline VsTest Task test run with two Test Cases associated to TestMethod.
Example mstest.log for MSTest.TestFramework v3.4.1
Example mstest.log for MSTest.TestFramework v2.2.3
NOTE: there are no MSTest code / Release pipeline changes between these two mstest.log examples. The only difference is the MSTest Nuget package version.
Expected behavior
AssemblyCleanup executes after all tests in the test run complete.
Actual behavior
AssemblyCleanup executes after the first test in a multi-test-point test run via Azure Devops Release pipeline VsTest Task when all tests are associated to the same TestMethod
Additional context
Azure DevOps Release pipeline definition Azure DevOps Release pipeline logs
This bug does not occur if all tests in the test run are associated to unique TestMethods. All tests in the test run must be associated to the same TestMethod for the bug to manifest.
This bug was already reported on Visual Studio Developer Community. I was told to raise an issue here.