'No test is available' in NUnit when mixing net6 and netframework472 test libraries in a single run

sbakharev commented 1 year ago

Description

Hello.

The Azure DevOps VsTest task executes tests from multiple found test dlls via a single vstest.console.exe call.

One interesting point is that the task allows you to use directory wildcards to search for test libraries, e.g. *\*.UnitTests.dll. So if the search directory has a structure like this:

TestsSearchDir
  TestDir1
    ...
    Module1.UnitTests.dll
    ...
  TestDir2
    ...
    Module2.UnitTests.dll
    ...

Then the vstest.console.exe call looks like this:

vstest.console.exe
  "%SEARCH_DIR%\TestDir1\Module1.UnitTests.dll"
  "%SEARCH_DIR%\TestDir2\Module2.UnitTests.dll"
  /TestAdapterPath:"%SEARCH_DIR%"

Notice the /TestAdapterPath argument which for some reason points to the search directory though in my pipeline (classic release) the parameter "Path to custom test adapters" is left blank. It's description states "Adapters residing in the same folder as the test assemblies are automatically discovered". So this task's behavior is probably a bug by itself but it has also uncovered a problem in vstest.console.exe.

When you run multiple test libraries with different target frameworks and the /TestAdapterPath points to their parent directory, then the netframework tests are reported as not found by NUnit and don't run.

Another detail is that if you filter all tests out with a /TestCaseFilter argument then the output would no longer say that there are no tests found in netframework libraries, but that all of the tests in all libraries do not match the filter criteria. So seems like it matters whether any tests have actually been executed.

/Parallel and /InIsolation arguments do not affect this behavior.

Steps to reproduce

Build the attached VsTestNUnitNoTestsRepro.zip solution using Debug configuration.

Run tests:

vstest.console.exe "%SOLUTION_DIR%\Net6TestLib1\bin\Debug\net6.0\Net6TestLib1.dll" "%SOLUTION_DIR%\NetFramework472TestLib1\bin\Debug\NetFramework472TestLib1.dll" /TestAdapterPath:"%SOLUTION_DIR%"

Expected behavior

Execute 1 test from Net6TestLib1
Execute 1 test from NetFramework472TestLib1

Actual behavior

1 test from Net6TestLib1 is executed
"No test is available" message is displayed about NetFramework472TestLib1

Starting test execution, please wait...
A total of 2 test files matched the specified pattern.
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in %SOLUTION_DIR%\Net6TestLib1\bin\Debug\net6.0\Net6TestLib1.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
  Passed Net6TestLib1_Test1 [29 ms]
No test is available in %SOLUTION_DIR%\NetFramework472TestLib1\bin\Debug\NetFramework472TestLib1.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 2,2805 Seconds

By the way it's also a significant problem that this test run is considered successful. This easily results in false positives during CI/CD executions. I've only found the problem by manually reviewing logs because I'm already cautious about different-framework runs after I've faced this similar issue some time ago.

Diagnostic logs

VsTestNUnitNoTestsRepro-VsTest-logs.zip

There we go, the net472 testhost uses the net6 test adapter (the first found via recursive search?):

TpTrace Verbose: 0 : 31520, 3, 2022/10/31, 16:35:03.757, 5160595953040, testhost.net472.exe, TestPluginCache.GetExtensionPaths: Added default extension paths: %SOLUTION_DIR%\Net6TestLib1\bin\Debug\net6.0\NUnit3.TestAdapter.dll

Environment

Windows 10
Test Execution Command Line Tool 17.3.2 (x64)
Visual Studio 17.3.6
NUnit 3.13.2
NUnit3TestAdapter 4.2.1

P.S. I've also retested this with the newest NUnit 3.13.3 and NUnit3TestAdapter 4.3.0 and the issue persists.

Evangelink commented 1 year ago

I think your issue relates to #3939. Could you please try the steps described there and let us know?

sbakharev commented 1 year ago

@Evangelink the issue you've referenced is about VSTest task test assemblies pattern and its default value. I've attached a whole reproduction solution with an example call to just vstest.console.exe (no VSTest task used) which does not use any default assemblies pattern - the test assemblies are explicitly listed in command line arguments. I've also attached the diagnostic logs which show the wrong test adapter being loaded for the net472 assembly. So these are seemingly different issues.

sbakharev commented 1 year ago

@Evangelink my issue is only connected to VSTest task due to the task's strange /TestAdapterPath argument setting behavior. But if the problem in vstest.console.exe would get fixed, the VSTest task would also stop failing in the described cases.

nohwnd commented 1 year ago

This is caused by azdo vstest task which sets the default adapter path to the root of the project. This is source of numerous issues, yet we are unable to the responsible team to remove it. https://github.com/microsoft/azure-pipelines-tasks/issues/16915

microsoft / vstest