RunFunctionalTests task failed to execute MSTestV2 tests in netcoreapp2 library

[GregoryOtt]

Environment

• TFS on-premises, provide the version: 16.122.27409.2 (Tfs2018.Update1)
• Private agent: 2.129.0 with VS 2017 15.6.1 (Windows Server 2016 Std - 10.0.14393 N/A Build 14393)
• Visual Studio Test Agent Deployment: 2.1.23 / Run Functional Tests: V1.0.55

Issue Description

Context

• Deploying the Visual Studio Test Agent on the build agent through the VS Test Agent Deployment task.

• Running netcoreapp2.0 librairy tests through RunFunctionalTests task in 'Test Assembly' mode.

• Referencing runsettings file with TargetFramewrok

  <RunSettings>
  <RunConfiguration>
  <TargetFrameworkersion>.NETCoreApp,Version=v2.0</TargetFrameworkersion>
  <TestAdaptersPaths>W:\Temp\XXXXTifTests\</TestAdaptersPaths>
  </RunConfiguration>
  </RunSettings>

  

• The XXXX.XXXX.IntegrationTests.dll contains a MSTest V2 extension of type RFC 003 defined in XXXX.XXXX.IntegrationTests.Core.TestCaseAssociationExtension.TestCaseTestMethodAttribute

• The XXXX.XXXX.IntegrationTests.dll contains one MSTest V2 test using my extension

• The class XXXX.XXXX.IntegrationTests.Core.TestCaseAssociationExtension.TestCaseTestMethodAttribute does not contain any test.

• I'm targeting netcoreapp2 because the lib references an ASP.NET Core 2 project

• Test runs fine with VSTest task

Expected behavior

Tests are ran

Actual behavior

The system failed to discover the test while loading my MSTest V2 extension.

Error logs

Task : Run Functional Tests Description : Run Coded UI/Selenium/Functional tests on a set of machines (using Test Agent) Version : 1.0.55 Author : Microsoft Corporation Help : More Information tf vc resolvePath $\XXXX\XXXX\Dev\Int\Src\XXXX.XXXX.IntegrationTests\BuildAndRelease.runsettings /loginType:OAuth /login:.,**** /noprompt W:\BUA06.4_w\1\s\Src\XXXX.XXXX.IntegrationTests\BuildAndRelease.runsettings Preparing task execution handler. Executing the powershell script: W:\BUA06.4_w_tasks\RunVisualStudioTestsusingTestAgent_d353d6a2-e361-4a8f-8d8c-123bebb71028\1.0.55\RunDistributedTests.ps1 PowerShellHandler.Execute - AddCommand(W:\BUA06.4_w_tasks\RunVisualStudioTestsusingTestAgent_d353d6a2-e361-4a8f-8d8c-123bebb71028\1.0.55\RunDistributedTests.ps1) PowerShellHandler.Execute - Add inputParameters PowerShellHandler.Execute - AddParameter(testMachineGroup=BUA06.XXXX.local) PowerShellHandler.Execute - AddParameter(dropLocation=W:\Temp\XXXXTifTests) PowerShellHandler.Execute - AddParameter(testSelection=testAssembly) PowerShellHandler.Execute - AddParameter(testPlan=14499) PowerShellHandler.Execute - AddParameter(testSuite=14501) PowerShellHandler.Execute - AddParameter(testConfiguration=6) PowerShellHandler.Execute - AddParameter(sourcefilters=XXXX.XXXX.IntegrationTests.dll) PowerShellHandler.Execute - AddParameter(testFilterCriteria=) PowerShellHandler.Execute - AddParameter(runSettingsFile=W:\BUA06.4_w\1\s\Src\XXXX.XXXX.IntegrationTests\BuildAndRelease.runsettings) PowerShellHandler.Execute - AddParameter(overrideRunParams=) PowerShellHandler.Execute - AddParameter(codeCoverageEnabled=false) PowerShellHandler.Execute - AddParameter(customSlicingEnabled=false) PowerShellHandler.Execute - AddParameter(testRunTitle=) PowerShellHandler.Execute - AddParameter(platform=$(BuildPlatform)) PowerShellHandler.Execute - AddParameter(configuration=$(BuildConfiguration)) PowerShellHandler.Execute - AddParameter(testConfigurations=) PowerShellHandler.Execute - AddParameter(autMachineGroup=) PowerShellHandler.Execute - Invoke Entering script RunDistributedTests.ps1 TestMachineGroup = BUA06.XXXX.local Test Drop Location = W:\Temp\XXXXTifTests Source Filter = XXXX.XXXX.IntegrationTests.dll Test Filter Criteria = RunSettings File = W:\BUA06.4_w\1\s\Src\XXXX.XXXX.IntegrationTests\BuildAndRelease.runsettings Build Platform = $(BuildPlatform) Build Configuration = $(BuildConfiguration) CodeCoverage Enabled = false TestRun Parameters to override = TestConfiguration = Application Under Test Machine Group = Loading module from path 'W:\BUA06.4\2.129.0\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Common\Microsoft.TeamFoundation.DistributedTask.Task.Common.psd1'. Loading module from path 'W:\BUA06.4\2.129.0\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Common\Microsoft.TeamFoundation.DistributedTask.Task.Common.psm1'. Loading module from path 'W:\BUA06.4\2.129.0\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.LegacySDK.dll'. Importing cmdlet 'Invoke-ResultPublisher'. Importing cmdlet 'Publish-TestResults'. Importing cmdlet 'Add-BuildArtifactLink'. Importing cmdlet 'Add-BuildAttachment'. Importing cmdlet 'Convert-String'. Importing cmdlet 'Copy-BuildArtifact'. Importing cmdlet 'Get-JavaDevelopmentKitPath'. Importing cmdlet 'Get-MSBuildLocation'. Importing cmdlet 'Get-ServiceEndpoint'. Importing cmdlet 'Get-TaskVariable'. Importing cmdlet 'Get-TfsClientCredentials'. Importing cmdlet 'Get-ToolPath'. Importing cmdlet 'Get-VisualStudioPath'. Importing cmdlet 'Get-VssConnection'. Importing cmdlet 'Invoke-Ant'. Importing cmdlet 'Invoke-BatchScript'. Importing cmdlet 'Invoke-Maven'. Importing cmdlet 'Invoke-MSBuild'. Importing cmdlet 'Invoke-Tool'. Importing cmdlet 'Invoke-VSTest'. Importing cmdlet 'Publish-BuildArtifact'. Importing cmdlet 'Register-XamarinLicense'. Importing cmdlet 'Unregister-XamarinLicense'. Importing cmdlet 'Get-X509Certificate'. Importing cmdlet 'Invoke-DeployTestAgent'. Importing cmdlet 'Invoke-RunDistributedTests'. Importing cmdlet 'Invoke-BlockEnvironment'. Importing cmdlet 'Copy-FilesToAzureBlob'. Importing cmdlet 'Copy-FilesToTargetMachine'. Importing cmdlet 'Copy-ToAzureMachines'. Importing cmdlet 'Remove-Environment'. Importing cmdlet 'Remove-EnvironmentResources'. Importing cmdlet 'Complete-EnvironmentOperation'. Importing cmdlet 'Complete-EnvironmentResourceOperation'. Importing cmdlet 'Complete-ResourceOperation'. Importing cmdlet 'Get-Environment'. Importing cmdlet 'Get-EnvironmentProperty'. Importing cmdlet 'Get-EnvironmentResources'. Importing cmdlet 'Get-ExternalIpAddress'. Importing cmdlet 'Get-ParsedSessionVariables'. Importing cmdlet 'Get-ProviderData'. Importing cmdlet 'Invoke-EnvironmentOperation'. Importing cmdlet 'Invoke-ResourceOperation'. Importing cmdlet 'Invoke-PsOnRemote'. Importing cmdlet 'New-OperationLog'. Importing cmdlet 'Copy-FilesToRemote'. Importing cmdlet 'Register-Environment'. Importing cmdlet 'Register-EnvironmentDefinition'. Importing cmdlet 'Register-Provider'. Importing cmdlet 'Register-ProviderData'. Importing cmdlet 'Invoke-UnblockEnvironment'. Importing cmdlet 'Add-TaskIssue'. Importing cmdlet 'Complete-Task'. Importing cmdlet 'Find-Files'. Importing cmdlet 'Get-LocalizedString'. Importing cmdlet 'Set-TaskProgress'. Importing cmdlet 'Set-TaskVariable'. Importing cmdlet 'Write-TaskDetail'. Importing cmdlet 'Enable-CodeCoverage'. Importing cmdlet 'Publish-CodeCoverage'. Exporting cmdlet 'Add-TaskIssue'. Exporting cmdlet 'Complete-Task'. Exporting cmdlet 'Find-Files'. Exporting cmdlet 'Get-LocalizedString'. Exporting cmdlet 'Set-TaskProgress'. Exporting cmdlet 'Set-TaskVariable'. Exporting cmdlet 'Write-TaskDetail'. Importing cmdlet 'Add-TaskIssue'. Importing cmdlet 'Complete-Task'. Importing cmdlet 'Find-Files'. Importing cmdlet 'Get-LocalizedString'. Importing cmdlet 'Set-TaskProgress'. Importing cmdlet 'Set-TaskVariable'. Importing cmdlet 'Write-TaskDetail'. Loading module from path 'W:\BUA06.4\2.129.0\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Internal\Microsoft.TeamFoundation.DistributedTask.Task.Internal.psd1'. Loading module from path 'W:\BUA06.4\2.129.0\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Internal\Microsoft.TeamFoundation.DistributedTask.Task.Internal.psm1'. Importing cmdlet 'Invoke-ResultPublisher'. Importing cmdlet 'Publish-TestResults'. Importing cmdlet 'Add-BuildArtifactLink'. Importing cmdlet 'Add-BuildAttachment'. Importing cmdlet 'Convert-String'. Importing cmdlet 'Copy-BuildArtifact'. Importing cmdlet 'Get-JavaDevelopmentKitPath'. Importing cmdlet 'Get-MSBuildLocation'. Importing cmdlet 'Get-ServiceEndpoint'. Importing cmdlet 'Get-TaskVariable'. Importing cmdlet 'Get-TfsClientCredentials'. Importing cmdlet 'Get-ToolPath'. Importing cmdlet 'Get-VisualStudioPath'. Importing cmdlet 'Get-VssConnection'. Importing cmdlet 'Invoke-Ant'. Importing cmdlet 'Invoke-BatchScript'. Importing cmdlet 'Invoke-Maven'. Importing cmdlet 'Invoke-MSBuild'. Importing cmdlet 'Invoke-Tool'. Importing cmdlet 'Invoke-VSTest'. Importing cmdlet 'Publish-BuildArtifact'. Importing cmdlet 'Register-XamarinLicense'. Importing cmdlet 'Unregister-XamarinLicense'. Importing cmdlet 'Get-X509Certificate'. Importing cmdlet 'Invoke-DeployTestAgent'. Importing cmdlet 'Invoke-RunDistributedTests'. Importing cmdlet 'Invoke-BlockEnvironment'. Importing cmdlet 'Copy-FilesToAzureBlob'. Importing cmdlet 'Copy-FilesToTargetMachine'. Importing cmdlet 'Copy-ToAzureMachines'. Importing cmdlet 'Remove-Environment'. Importing cmdlet 'Remove-EnvironmentResources'. Importing cmdlet 'Complete-EnvironmentOperation'. Importing cmdlet 'Complete-EnvironmentResourceOperation'. Importing cmdlet 'Complete-ResourceOperation'. Importing cmdlet 'Get-Environment'. Importing cmdlet 'Get-EnvironmentProperty'. Importing cmdlet 'Get-EnvironmentResources'. Importing cmdlet 'Get-ExternalIpAddress'. Importing cmdlet 'Get-ParsedSessionVariables'. Importing cmdlet 'Get-ProviderData'. Importing cmdlet 'Invoke-EnvironmentOperation'. Importing cmdlet 'Invoke-ResourceOperation'. Importing cmdlet 'Invoke-PsOnRemote'. Importing cmdlet 'New-OperationLog'. Importing cmdlet 'Copy-FilesToRemote'. Importing cmdlet 'Register-Environment'. Importing cmdlet 'Register-EnvironmentDefinition'. Importing cmdlet 'Register-Provider'. Importing cmdlet 'Register-ProviderData'. Importing cmdlet 'Invoke-UnblockEnvironment'. Importing cmdlet 'Add-TaskIssue'. Importing cmdlet 'Complete-Task'. Importing cmdlet 'Find-Files'. Importing cmdlet 'Get-LocalizedString'. Importing cmdlet 'Set-TaskProgress'. Importing cmdlet 'Set-TaskVariable'. Importing cmdlet 'Write-TaskDetail'. Importing cmdlet 'Enable-CodeCoverage'. Importing cmdlet 'Publish-CodeCoverage'. Exporting cmdlet 'Add-BuildArtifactLink'. Exporting cmdlet 'Add-BuildAttachment'. Exporting cmdlet 'Convert-String'. Exporting cmdlet 'Copy-BuildArtifact'. Exporting cmdlet 'Get-JavaDevelopmentKitPath'. Exporting cmdlet 'Get-MSBuildLocation'. Exporting cmdlet 'Get-ServiceEndpoint'. Exporting cmdlet 'Get-TaskVariable'. Exporting cmdlet 'Get-TfsClientCredentials'. Exporting cmdlet 'Get-ToolPath'. Exporting cmdlet 'Get-VisualStudioPath'. Exporting cmdlet 'Get-VssConnection'. Exporting cmdlet 'Invoke-Ant'. Exporting cmdlet 'Invoke-BatchScript'. Exporting cmdlet 'Invoke-Maven'. Exporting cmdlet 'Invoke-MSBuild'. Exporting cmdlet 'Invoke-Tool'. Exporting cmdlet 'Invoke-VSTest'. Exporting cmdlet 'Publish-BuildArtifact'. Exporting cmdlet 'Register-XamarinLicense'. Exporting cmdlet 'Unregister-XamarinLicense'. Exporting cmdlet 'Get-X509Certificate'. Importing cmdlet 'Add-BuildArtifactLink'. Importing cmdlet 'Add-BuildAttachment'. Importing cmdlet 'Convert-String'. Importing cmdlet 'Copy-BuildArtifact'. Importing cmdlet 'Get-JavaDevelopmentKitPath'. Importing cmdlet 'Get-MSBuildLocation'. Importing cmdlet 'Get-ServiceEndpoint'. Importing cmdlet 'Get-TaskVariable'. Importing cmdlet 'Get-TfsClientCredentials'. Importing cmdlet 'Get-ToolPath'. Importing cmdlet 'Get-VisualStudioPath'. Importing cmdlet 'Get-VssConnection'. Importing cmdlet 'Get-X509Certificate'. Importing cmdlet 'Invoke-Ant'. Importing cmdlet 'Invoke-BatchScript'. Importing cmdlet 'Invoke-Maven'. Importing cmdlet 'Invoke-MSBuild'. Importing cmdlet 'Invoke-Tool'. Importing cmdlet 'Invoke-VSTest'. Importing cmdlet 'Publish-BuildArtifact'. Importing cmdlet 'Register-XamarinLicense'. Importing cmdlet 'Unregister-XamarinLicense'. Loading module from path 'W:\BUA06.4\2.129.0\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.DTA\Microsoft.TeamFoundation.DistributedTask.Task.DTA.psm1'. Importing cmdlet 'Invoke-ResultPublisher'. Importing cmdlet 'Publish-TestResults'. Importing cmdlet 'Add-BuildArtifactLink'. Importing cmdlet 'Add-BuildAttachment'. Importing cmdlet 'Convert-String'. Importing cmdlet 'Copy-BuildArtifact'. Importing cmdlet 'Get-JavaDevelopmentKitPath'. Importing cmdlet 'Get-MSBuildLocation'. Importing cmdlet 'Get-ServiceEndpoint'. Importing cmdlet 'Get-TaskVariable'. Importing cmdlet 'Get-TfsClientCredentials'. Importing cmdlet 'Get-ToolPath'. Importing cmdlet 'Get-VisualStudioPath'. Importing cmdlet 'Get-VssConnection'. Importing cmdlet 'Invoke-Ant'. Importing cmdlet 'Invoke-BatchScript'. Importing cmdlet 'Invoke-Maven'. Importing cmdlet 'Invoke-MSBuild'. Importing cmdlet 'Invoke-Tool'. Importing cmdlet 'Invoke-VSTest'. Importing cmdlet 'Publish-BuildArtifact'. Importing cmdlet 'Register-XamarinLicense'. Importing cmdlet 'Unregister-XamarinLicense'. Importing cmdlet 'Get-X509Certificate'. Importing cmdlet 'Invoke-DeployTestAgent'. Importing cmdlet 'Invoke-RunDistributedTests'. Importing cmdlet 'Invoke-BlockEnvironment'. Importing cmdlet 'Copy-FilesToAzureBlob'. Importing cmdlet 'Copy-FilesToTargetMachine'. Importing cmdlet 'Copy-ToAzureMachines'. Importing cmdlet 'Remove-Environment'. Importing cmdlet 'Remove-EnvironmentResources'. Importing cmdlet 'Complete-EnvironmentOperation'. Importing cmdlet 'Complete-EnvironmentResourceOperation'. Importing cmdlet 'Complete-ResourceOperation'. Importing cmdlet 'Get-Environment'. Importing cmdlet 'Get-EnvironmentProperty'. Importing cmdlet 'Get-EnvironmentResources'. Importing cmdlet 'Get-ExternalIpAddress'. Importing cmdlet 'Get-ParsedSessionVariables'. Importing cmdlet 'Get-ProviderData'. Importing cmdlet 'Invoke-EnvironmentOperation'. Importing cmdlet 'Invoke-ResourceOperation'. Importing cmdlet 'Invoke-PsOnRemote'. Importing cmdlet 'New-OperationLog'. Importing cmdlet 'Copy-FilesToRemote'. Importing cmdlet 'Register-Environment'. Importing cmdlet 'Register-EnvironmentDefinition'. Importing cmdlet 'Register-Provider'. Importing cmdlet 'Register-ProviderData'. Importing cmdlet 'Invoke-UnblockEnvironment'. Importing cmdlet 'Add-TaskIssue'. Importing cmdlet 'Complete-Task'. Importing cmdlet 'Find-Files'. Importing cmdlet 'Get-LocalizedString'. Importing cmdlet 'Set-TaskProgress'. Importing cmdlet 'Set-TaskVariable'. Importing cmdlet 'Write-TaskDetail'. Importing cmdlet 'Enable-CodeCoverage'. Importing cmdlet 'Publish-CodeCoverage'. Exporting cmdlet 'Invoke-DeployTestAgent'. Exporting cmdlet 'Invoke-RunDistributedTests'. Importing cmdlet 'Invoke-DeployTestAgent'. Importing cmdlet 'Invoke-RunDistributedTests'. Getting the connection object Invoke - Get-VssConnection cmdlet Get - SecretService and EndpointService from TaskContext Fetched Endpoint:SystemVssConnection AuthorizationType : OAuth VssConnection created UnregisterTestAgent script Path = Calling Invoke-RunDistributedTests Invoking Run Distributed Tests with Register Environment support DistributedTests: Environment WinRm Protocol HTTP. DistributedTests: Definition Name : XXXX.Tif DistributedTests: Build Name : 5510 DistributedTests: Build Id : 5510 DistributedTests: Run Settings File Path : W:\BUA06.4_w\1\s\Src\XXXX.XXXX.IntegrationTests\BuildAndRelease.runsettings DistributedTests: Creating test settings. test settings name : a48f31d9-3dea-4a59-bc9a-ca2db9146e69 DistributedTests: Done creating test settings. test settings Id : 1472 DistributedTests: Creating run for selected test assemblies with following parameters DistributedTests: SourceFilter: XXXX.XXXX.IntegrationTests.dll TestCaseFilter: DistributedTests: Run title: TestRun_XXXX.Tif_5510 DistributedTests: is automated: True DistributedTests: Test Machines Environment Url: dta://env/f91ab305-0ec6-4632-bb83-002ffdbcdcd6/_apis//201803121513126543 DistributedTests: test settings id : 1472 DistributedTests: build location: W:\Temp\XXXXTifTests DistributedTests: build id: 5510 DistributedTests: test configuration mapping: DistributedTests: Test Run with Id 5736 Queued DistributedTests: Test run '5736' is in 'InProgress' state. DistributedTests: Test run '5736' is in 'InProgress' state. DistributedTests: Test run '5736' is in 'InProgress' state. DistributedTests: Test run '5736' is in 'InProgress' state. DistributedTests: Test run '5736' is in 'Aborted' state. DistributedTests: Test run is aborted. Logging details of the run logs. Processed: ##vso[task.logissue type=warning]DistributedTests: Test run is aborted. Logging details of the run logs. DistributedTests: New test run created. Processed: ##vso[task.logissue type=warning]DistributedTests: New test run created. Test Run queued for Project Collection Build Service (SSI). DistributedTests: Test discovery started. Processed: ##vso[task.logissue type=warning]DistributedTests: Test discovery started. DistributedTests: Test Run Discovery Aborted . Test run id : 5736 Processed: ##vso[task.logissue type=warning]DistributedTests: Test Run Discovery Aborted . Test run id : 5736 DistributedTests: UnExpected error occurred during test execution. Try again. Processed: ##vso[task.logissue type=warning]DistributedTests: UnExpected error occurred during test execution. Try again. DistributedTests: Error : [MSTest][Discovery][W:\Temp\XXXXTifTests\XXXX.XXXX.IntegrationTests.dll] MSTestAdapter failed to discover tests in class 'XXXX.XXXX.IntegrationTests.Core.TestCaseAssociationExtension.TestCaseTestMethodAttribute' of assembly 'W:\Temp\XXXXTifTests\XXXX.XXXX.IntegrationTests.dll' because Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.. Processed: ##vso[task.logissue type=warning]DistributedTests: Error : [MSTest][Discovery][W:\Temp\XXXXTifTests\XXXX.XXXX.IntegrationTests.dll] MSTestAdapter failed to discover tests in class 'XXXX.XXXX.IntegrationTests.Core.TestCaseAssociationExtension.TestCaseTestMethodAttribute' of assembly 'W:\Temp\XXXXTifTests\XXXX.XXXX.IntegrationTests.dll' because Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.. DistributedTests: Test run aborted. Test run id: 5736 Processed: ##vso[task.logissue type=warning]DistributedTests: Test run aborted. Test run id: 5736 System.Exception: The test run was aborted, failing the task. Processed: ##vso[task.logissue type=error;]System.Exception: The test run was aborted, failing the task. DistributedTests: Deleted test settings with id : 1472 PowerShell script completed with 1 errors.

[GregoryOtt]

As far as I understand, the 'Run Functional Tests' task is(will be) deprecated. I've not seen that the 'Visual Studio Test' task is now able to test against a Test Plan which is my final goal.

I close this issue and reopen for the VSTest task. Sorry for the noise.

[schmidtaa]

@GregoryOtt Do you have VSTest issue in github for running tests against Test Plan?

[GregoryOtt]

@schmidtaa not in VSTest repo. In this repo I have a more recent issue => #6674

[schmidtaa]

Got it, thanks.