search

xunit / visualstudio.xunit

VSTest runner for xUnit.net (for Visual Studio Test Explorer and dotnet test)
https://xunit.net/
Other
141 stars 81 forks source link

xunit.runner.visualstudio 2.8.1 fails to discover .Net 4.7.2 Unit Tests #409

Open onur-ozguzel opened 2 months ago

onur-ozguzel commented 2 months ago

Hello,

For a long time, we were using xunit.runner.visualstudio 2.4.5. vstest.console.exe was able to detect both our .NET 6.0 and .NET Framework 4.7.2 unit tests with these parameters:

vstest.console.exe "C:\a\2\s\" "C:\a\2\s\" "C:\a\2\s\" /TestCaseFilter:"Category=UnitTest" /Settings:"C:\a_temp\lbxw3zg1gnt.tmp.runsettings" /Logger:"trx" /TestAdapterPath:"C:\a\2\s" /Framework:.NETFramework,Version=v4.7.2

After we upgraded to xunit.runner.visualstudio 2.8.1, our .NET Framework unit test discoveries stopped working.

We tested all the new versions, the results were the same except 2.5.0. So discovery were not working correctly for 2.8.1 - 2.8.0 - 2.5.8 - 2.5.7 - 2.5.6 - 2.5.5 - 2.5.4 - 2.5.3 - 2.5.1.

The error message we were getting was:

No test matches the given testcase filter Category=UnitTest in "C:\a\2\s\" No test matches the given testcase filter Category=UnitTest in "C:\a\2\s\" No test matches the given testcase filter Category=UnitTest in "C:\a\2\s\"

For discovery to work again for .NET 4.7.2 Unit Tests, we had to change TestAdapterPath like below:

vstest.console.exe "C:\a\2\s\" C:\a\2\s\" C:\a\2\s\" /TestCaseFilter:"Category=UnitTest" /Settings:"C:\a_temp\qfap23dg5er.tmp.runsettings" /Logger:"trx" /TestAdapterPath:"C:\a\2\s\" /Framework:.NETFramework,Version=v4.7.2

We have our project in C:\a\2\s folder. Meaning inside this folder, we have both .NET 6.0 and .NET Framework 4.7.2 Unit Tests and Test adapter assemblies. Our hunch is, after xunit.runner.visualstudio 2.4.5 and 2.5.0, vstest.console.exe started to find the wrong test adapter for .NET Framework 4.7.2 tests if we provide the base path, and because of that it failed to discover them.

In the newer versions of xunit.runner.visualstudio, can we provide the base path of the TestAdapterPath and make it work just like it was working in 2.4.5 and 2.5.0?

PS: For .NET 6.0 unit test, it is still able to discover tests by providing the base path to the TestAdapterPath like below:

vstest.console.exe "C:\a\2\s\" "C:\a\2\s\" "C:\a\2\s\" /TestCaseFilter:"Category=UnitTest" /Settings:"C:\a_temp\wizpx4ngr2k.tmp.runsettings" /Logger:"trx" /TestAdapterPath:"C:\a\2\s" /Framework:.NETCoreApp,Version=v6.0

bradwilson commented 2 months ago

Our hunch is, after xunit.runner.visualstudio 2.4.5 and 2.5.0, vstest.console.exe started to find the wrong test adapter for .NET Framework 4.7.2 tests if we provide the base path, and because of that it failed to discover them.

I suspect your hunch is correct, in which case the bug is in VSTest and not with us, because we don't control that directory-based adapter discovery logic.

I don't see much that changed between 2.5.0 and 2.5.1 that looks "suspect"; it was mostly pulling newer versions of xunit.runner.utility from the core project. If the VSTest wants to try to narrow it down on our side, we offered a few pre-release 2.5.1 packages that might help them.