dotnet test on linux fails to find tests

[mjkkirschner]

Description

Unclear if this is an Nunit or dotnet test issue, but starting here as Nunit seems to report the correct number of tests.

When running under linux - ubuntu 22.04 locally or on github hosted runner - the command: dotnet test pathTotestBinary.dll fails to run any tests. This binary references nunit3. The nunit framework, engine, and test adapter binaries are in the same directory as the test binary.

Interestingly, in verbose mode, nunit test adapter / executer reports that it finds all the tests.

Also -t list mode shows all the tests.

This same command works fine on windows.

dotnet 6.0.406 is installed locally.

output looks like:

parallels@parallels-Parallels-Virtual-Platform:~/Documents/Dynamo$ dotnet test bin/NET60_Linux/Release/DynamoCoreTests.dll -v:d --diag:dotnettestlog.txt
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Logging Vstest Diagnostics in file: /home/parallels/Documents/Dynamo/dotnettestlog.txt
A total of 1 test files matched the specified pattern.
/home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll
Logging TestHost Diagnostics in file: /home/parallels/Documents/Dynamo/dotnettestlog.host.23-02-27_19-52-44_18198_5.txt
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll
   NUnit3TestExecutor discovered 2568 of 2568 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
No test is available in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Steps to reproduce

dotnet 6 / 7 nunit 3 adapter 4.2.1 nunit 13.3

Expected behavior

~2500 tests should be found

Actual behavior

dotnet test reports there are not tests in the assembly.

Diagnostic logs

dotnettestlog.zip

Environment

using parallels locally, but issue reproduces on github ubuntu runner.

[mjkkirschner]

I've noticed that this same message is sometimes shown when test dependencies are missing (like Moq) is that expected?

[Evangelink]

Hi @mjkkirschner,

It would be good to have access to the project configuration to be able to tell for sure but from the logs it would seem to be an issue with NUnit.

TpTrace Information: 0 : 27309, 4, 2023/02/27, 19:52:34.684, 4441635620303, testhost.dll, Running all tests in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:34.684, 4441635972496, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":7,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"Running all tests in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll"}}
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:34.686, 4441637991471, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestRunMessageLoggerProxy.SendMessage: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.OnTestRunMessage, took 2 ms.
TpTrace Information: 0 : 27309, 4, 2023/02/27, 19:52:35.162, 4442113309016, testhost.dll,    NUnit3TestExecutor discovered 2568 of 2568 NUnit test cases using Current Discovery mode, Non-Explicit run
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.162, 4442113752886, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":7,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"   NUnit3TestExecutor discovered 2568 of 2568 NUnit test cases using Current Discovery mode, Non-Explicit run"}}
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.164, 4442115584495, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestRunMessageLoggerProxy.SendMessage: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.OnTestRunMessage, took 2 ms.
TpTrace Information: 0 : 27309, 4, 2023/02/27, 19:52:35.218, 4442170229525, testhost.dll, NUnit Adapter 4.2.0.0: Test execution complete
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.219, 4442170736460, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":7,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"NUnit Adapter 4.2.0.0: Test execution complete"}}
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.220, 4442171334186, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestRunMessageLoggerProxy.SendMessage: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.OnTestRunMessage, took 1 ms.
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.231, 4442182347272, testhost.dll, BaseRunTests.RunTestInternalWithExecutors: Completed running tests for executor://NUnit3TestExecutor
TpTrace Warning: 0 : 27309, 4, 2023/02/27, 19:52:35.232, 4442183454544, testhost.dll, No test is available in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

As you can see, the testhost is starting the test execution, then NUnit says it found some tests an then says it's done invoking them but the test platform didn't receive any test result hence the message.

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 10 days.

[mjkkirschner]

Hi @Evangelink

thanks for the reply, to reproduce the issue you can checkout this github action: https://github.com/DynamoDS/Dynamo/blob/master/.github/workflows/dynamoNet6.0_linux_build.yml note that the tests are currently disabled due to the above issue.

the test assembly csproj: https://github.com/DynamoDS/Dynamo/blob/master/test/DynamoCoreTests/DynamoCoreTests.csproj

Please let me know if that was not what you meant by access to the project configuration.

[nohwnd]

This was hopefully solved in the meantime, we don't have others reporting this problem. If not let me know.