Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria when run parametrised associated test from test plan

[ghostinside]

Note

when trying to run parametrized associated with test case test

[Test, TestCaseSource(typeof(TestSource))] public virtual void CheckTitleTest(Uri baseUrl) { var arenaInstance = new Arena(baseUrl); Uri homeUrl = arenaInstance.GetHomeUrl(); OpenPage(homeUrl); arenaInstance .AllGamesPage .CheckPageTitleVisible(); }

got the error that test could not be found (please see attached logs), if execute test like this (without parameters, test would be run successfully )

[Test, TestCaseSource(typeof(TestSource))] public virtual void CheckTitleTest() { var arenaInstance = new Arena(TestContext.Parameters["URL"]); Uri homeUrl = arenaInstance.GetHomeUrl(); OpenPage(homeUrl); arenaInstance .AllGamesPage .CheckPageTitleVisible(); }

Error logs

2019-10-21T18:10:42.3532229Z ##[section]Starting: Run tests 2019-10-21T18:10:42.3867082Z ============================================================================== 2019-10-21T18:10:42.3867803Z Task : Visual Studio Test 2019-10-21T18:10:42.3867954Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2). 2019-10-21T18:10:42.3868058Z Version : 2.157.5 2019-10-21T18:10:42.3868143Z Author : Microsoft Corporation 2019-10-21T18:10:42.3868268Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest 2019-10-21T18:10:42.3868356Z ============================================================================== 2019-10-21T18:10:44.5329180Z SystemVssConnection exists true 2019-10-21T18:10:44.7481202Z SystemVssConnection exists true 2019-10-21T18:10:44.9267352Z SystemVssConnection exists true 2019-10-21T18:10:45.0617309Z In distributed testing flow 2019-10-21T18:10:45.0617554Z ====================================================== 2019-10-21T18:10:45.0618492Z Test selector : Test run 2019-10-21T18:10:45.0618783Z Test run Id : '1039606' 2019-10-21T18:10:45.0619051Z Search folder : C:\vsts-agent_work\r1441\a 2019-10-21T18:10:45.0622450Z VisualStudio version selected for test execution : latest 2019-10-21T18:10:45.0623230Z Attempting to find vstest.console from a visual studio installation with version [16.0,17.0). 2019-10-21T18:10:45.3451718Z Attempting to find vstest.console from a visual studio build tools installation with version [16.0,17.0). 2019-10-21T18:10:45.4243580Z Distributed test execution, number of agents in job : 1 2019-10-21T18:10:45.4319001Z Run in parallel : false 2019-10-21T18:10:45.4326232Z Run in isolation : false 2019-10-21T18:10:45.4328581Z Path to custom adapters : null 2019-10-21T18:10:45.4348360Z Other console options : null 2019-10-21T18:10:45.4358213Z Code coverage enabled : false 2019-10-21T18:10:45.4373418Z Diagnostics enabled : true 2019-10-21T18:10:45.4773728Z ====================================================== 2019-10-21T18:10:45.4775378Z Source filter: *test*.dll,!*\TestAdapter.dll,!\obj* 2019-10-21T18:10:45.6634484Z SystemVssConnection exists true 2019-10-21T18:10:45.6843657Z [command]C:\vsts-agent_work_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.157.5\Modules\DTAExecutionHost.exe --inputFile C:\vsts-agent_work_temp\input_1a2f9630-f42e-11e9-83ca-6dda2ed3b348.json 2019-10-21T18:10:46.4485621Z ########################################################################## 2019-10-21T18:10:46.4494491Z DtaExecutionHost version 17.157.29404.1. 2019-10-21T18:10:49.1306660Z =========================================== 2019-10-21T18:10:49.1307062Z AgentName: NY-VSOBUILD02-NY-VSOBUILD02-15 2019-10-21T18:10:49.1307412Z ServiceUrl: ### 2019-10-21T18:10:49.1307619Z TestPlatformVersion: 16.0.1 2019-10-21T18:10:49.1307775Z EnvironmentUri: vstest://env/Department_Base_Engineers/_apis/release/2/27/27/1 2019-10-21T18:10:49.1307891Z QueryForTaskIntervalInMilliseconds: 3000 2019-10-21T18:10:49.1308070Z MaxQueryForTaskIntervalInMilliseconds: 10000 2019-10-21T18:10:49.1308183Z QueueNotFoundDelayTimeInMilliseconds: 3000 2019-10-21T18:10:49.1308289Z MaxQueueNotFoundDelayTimeInMilliseconds: 50000 2019-10-21T18:10:49.1308396Z =========================================== 2019-10-21T18:10:50.2704319Z TestExecutionHost.Execute: Registered TestAgent : 23 : NY-VSOBUILD02-NY-VSOBUILD02-15 2019-10-21T18:10:50.3781957Z Updated Run Settings: 2019-10-21T18:10:50.3814337Z 2019-10-21T18:10:50.3814818Z
2019-10-21T18:10:50.3815047Z 2019-10-21T18:10:50.3815289Z 1000 2019-10-21T18:10:50.3815619Z C:\vsts-agent_work_temp\TestResults 2019-10-21T18:10:50.3820229Z 2019-10-21T18:10:50.3820471Z 2019-10-21T18:10:50.3821400Z 2019-10-21T18:10:50.3821738Z 2019-10-21T18:10:50.3822001Z 2019-10-21T18:10:50.3822278Z C:\vsts-agent_work_temp\TestResults 2019-10-21T18:10:50.3822528Z 2019-10-21T18:10:50.3822816Z 2019-10-21T18:10:50.3823075Z 2019-10-21T18:10:50.3823297Z 2019-10-21T18:10:50.3823560Z 2019-10-21T18:10:50.3823830Z 2019-10-21T18:10:50.3824072Z 2019-10-21T18:10:50.3824305Z 2019-10-21T18:10:50.3824515Z 2019-10-21T18:10:50.3824725Z 2019-10-21T18:10:50.3825085Z 2019-10-21T18:10:50.4200854Z IsValidServiceResponse: Received None command..Service Workflow is not active 2019-10-21T18:10:50.6409720Z Creating run for selected test assemblies with following parameters 2019-10-21T18:10:50.6412379Z SourceFilter: .dll TestCaseFilter: 2019-10-21T18:10:50.6415546Z Run title: TestRun_TestAutoSeleniumTestCase_Release-17 2019-10-21T18:10:50.6418093Z Build location: C:\vsts-agent_work\r1441\a 2019-10-21T18:10:50.6420544Z Build Id: 95809 2019-10-21T18:10:51.1098238Z Test run with Id 1039606 associated 2019-10-21T18:11:01.5567067Z Received the command : Start 2019-10-21T18:11:01.5667371Z TestExecutionHost.ProcessCommand. Start Command handled 2019-10-21T18:11:01.8240933Z Slice with id = 22, of type = 'Execution' received. 2019-10-21T18:11:02.1513780Z Count of test sources found: 1 2019-10-21T18:11:02.2413128Z ================================================================= 2019-10-21T18:11:02.2415961Z Discovering tests from sources 2019-10-21T18:11:02.2528763Z Using new test platform for test execution 2019-10-21T18:11:06.2695717Z DiscoveryMessage : NUnit Adapter 3.15.1.0: Test discovery starting 2019-10-21T18:11:08.0939224Z DiscoveryMessage : NUnit Adapter 3.15.1.0: Test discovery complete 2019-10-21T18:11:08.2930040Z Number of testcases discovered : 6 2019-10-21T18:11:08.3878282Z Discovered tests 6 from sources 2019-10-21T18:11:08.3878686Z ================================================================= 2019-10-21T18:11:08.4857605Z [RunStatistics]This execution slice with id '22', received '2' testcases to execute out of which '0' is discovered. 2019-10-21T18:11:08.6320700Z ##[error]The slice of type 'Execution' is 'Aborted' because of the error : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct. 2019-10-21T18:11:08.6341459Z at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModdelContext, CancellationToken cancellationToken) 2019-10-21T18:11:08.6341772Z at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken) 2019-10-21T18:11:22.7098232Z Received the command : Stop 2019-10-21T18:11:22.7098602Z TestExecutionHost.ProcessCommand. Stop Command handled 2019-10-21T18:11:22.7116445Z SliceFetch Aborted. Moving to the TestHostEnd phase 2019-10-21T18:11:22.9504502Z Please use this link to analyze the test run : ###/_TestManagement/Runs#_a=resultQuery&runId=1039606&queryPath=Recent+Run%2FRun1039606 2019-10-21T18:11:22.9506246Z Test run '1039606' is in 'Aborted' state with 'Total Tests' : 2 and 'Passed Tests' : 0. 2019-10-21T18:11:22.9567894Z ##[error]Test run is aborted. Logging details of the run logs. 2019-10-21T18:11:22.9572976Z ##[error]System.Exception: The test run was aborted, failing the task. 2019-10-21T18:11:23.1882511Z ########################################################################## 2019-10-21T18:11:23.2842652Z ##[section]Finishing: Run tests

[ShreyasRmsft]

@ghostinside please provide the full zip logs obtained by clicking on "download all logs" with system.debug = true.

[ghostinside]

@ShreyasRmsft here you are

ReleaseLogs_30.zip

[ShreyasRmsft]

Thanks, taking a look

[ShreyasRmsft]

I see you did not provide all the logs.

There are diagnostic logs from the test platform that get uploaded. Something along the lines of Slice-26_3wqkse.host.19-10-23_04-19-15_85586_6.diag

Please make sure you provide them too.

[ghostinside]

@ShreyasRmsft how can i get them? just downloaded from the root

[ghostinside]

[ShreyasRmsft]

Hmm, I think i have an inkling as to why these did not get uploaded. Can you please use the latest version of the test platform using https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer?view=azure-devops

And then provide the logs. There was a bug in the older versions of the platform that resulted in a lock on to these files.

[ghostinside]

@ShreyasRmsft latest stable, right?

[ghostinside]

@ShreyasRmsft please find in attachment

ReleaseLogs_31.zip

[ghostinside]

@ShreyasRmsft any news?

[ShreyasRmsft]

@ghostinside we do not most of the tasks advanced features (re-run, distribution, tia) for nunit and xunit data driven tests as of today.

The reason being the nunit and xunit adapters behave a little weird when it comes to discovering and execution data driven tests. They put the display name of the test as the FQD but when your run the list fully qualified name command of vstest of the very same tests it gives a different name (does not include the params expanded). This is evident in the diag logs from the platform.

Because of this we are unable to uniquely identify those tests when the same test cases are sent back from the server to execute on different agents.

We have it in our back log to reach out to the nunit and xunit folks to understand what it would take to fix this but until then please note that distribution, rerun and tia will not work with nunit and xunit data driven tests.

[ghostinside]

@ShreyasRmsft can we speedup this fix somehow? Using test case source i can run one test several times natively

[ShreyasRmsft]

@ghostinside the fix might not be at our end and might be in the nunit and xunit adapters. I will keep you updated.

[ShreyasRmsft]

@ghostinside we've added some support for running certain types of xunit data driven tests in distributed mode, please give it a try now and let us know. PS: We still do not support running nunit datadriven tests in distributed mode or with some advanced features of the task.

[rfancher-nanostring]

I'm encountering this issue as well. It would be really great if this type of execution was supported. xUnit is very common and we are using basic xUnit tests to populate the test results from other tools into ADO - except this one issue seems to be blocking.

Here are the logs, can we please get this re-opened and fixed?

2023-07-06T18:41:39.7936808Z ##[debug] Populating TCM Context Data 2023-07-06T18:41:39.8207142Z ##[debug]TCMDataProcessor.ProcessSettings: Obtained TcmTestProperties - {"BuildConfigurationId":74324,"BuildDirectory":"C:\RanorexVerification\RanorexXunit\RanorexXunit\","BuildFlavor":"","BuildNumber":"1.3.0.78","BuildPlatform":"","BuildUri":"vstfs:\/\/\/Build\/Build\/74324","TeamProject":"Gemini","TestCases":[{"FullyQualifiedName":"Gemini_Ranorex_137970","Source":"RanorexVerification.dll","TestCaseId":137970,"TestConfigurationId":7,"TestConfigurationName":"Windows 10","TestPointId":67153}],"TestPlanId":108133,"TestRunId":262897,"TfsServerCollectionUrl":"https:\/\/dev.azure.com\/Nanostring\/"} 2023-07-06T18:41:39.8328872Z ##[debug]TCMDataProcessor.ProcessSettings: Adding param to settings for tcm context json filePath - C:\Users\ranorexAgentAdmin\AppData\Local\Temp\TcmTestProperties-e20765fa-f7c7-4c0e-81e8-6abdd4e0d588.props 2023-07-06T18:41:39.8393163Z ##[debug]DiscoverTests.PerformDiscovery : Performing discovery for tests 2023-07-06T18:41:39.8398030Z ##[debug]Initializing V2 test case filter. 2023-07-06T18:41:39.8399026Z ##[debug]TestCaseFilter.TestCaseFilter : Creating new Filter instance. 2023-07-06T18:41:39.8399886Z ##[debug]TestCaseFilter.Initialize : Initializing new filter instance with the filter string. 2023-07-06T18:41:39.8400990Z ##[debug]Probing the DLL: Microsoft.VisualStudio.TestPlatform.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a from paths: c:\Agent_work_tool\VsTest\17.3.2\x64\tools\net451\Common7\IDE\Extensions\TestPlatform 2023-07-06T18:41:39.8413808Z ##[debug]PERF: TestPlatformAssemblyLoader:CurrentDomainAssemblyResolver_TestExtensions: took 2.5481 ms 2023-07-06T18:41:39.8419089Z ##[debug]SourceFilterFactory.GetSourceFilter() MiniMatchBasedTestSourcesFile is provided, will return MiniMatchSourceFilter 2023-07-06T18:41:39.8420127Z ##[debug]MiniMatchSourceFilter.PrependRecursiveFilter() Filter string modified to **\RanorexVerification.dll 2023-07-06T18:41:39.8446409Z Count of test sources found: 0 2023-07-06T18:41:39.8931203Z ##[warning]No test sources found 2023-07-06T18:41:39.8933099Z ##[debug]Processed: ##vso[task.logissue type=warning;]No test sources found 2023-07-06T18:41:39.8935832Z ##[debug]PERF: DiscoverTests.PerformDiscovery: took 57.2888 ms 2023-07-06T18:41:39.9451820Z [RunStatistics]This execution slice with id '65', received '1' testcases to execute out of which '0' is discovered. 2023-07-06T18:41:39.9453017Z ##[debug]PERF: Execution:FetchPreviousPhaseData: took 117.5093 ms 2023-07-06T18:41:39.9482708Z ##[debug]Initializing V2 Test Logger manager. 2023-07-06T18:41:39.9486663Z List of test cases in slice with id: 65 2023-07-06T18:41:39.9487835Z Source: RanorexVerification.dll, FullyQualifiedTestName: Gemini_Ranorex_137970 2023-07-06T18:41:39.9488615Z ##[debug]PERF: Execution:Run: took 2.3094 ms 2023-07-06T18:41:39.9992021Z ##[error]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct. 2023-07-06T18:41:39.9993699Z ##[debug]Processed: ##vso[task.logissue type=error;]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct. 2023-07-06T18:41:39.9995448Z at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken) 2023-07-06T18:41:39.9998064Z at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)