MSTestAdapter failed to discover tests because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' does not have an implementation

Shanayara commented 2 years ago

Description

We've long had a running VSTest pipeline for our projects unit tests, which target Framework 4.8 and run MSTest.

The YAML of the unit test on DevOps is very minimal:

- name: DebugBuildFolder
  default: '$(DebugBuildFolder)'
- name: DebugBuildArtifactName
  default: '$(DebugBuildArtifactName)'

steps:
  - task: DownloadPipelineArtifact@2
    displayName: 'Download ${{ parameters.DebugBuildArtifactName }} artifact'
    inputs:
      buildType: 'current'
      artifactName: ${{ parameters.DebugBuildArtifactName }}
      targetPath: '$(Build.SourcesDirectory)'
  - task: VSTest@2
    displayName: 'UnitTests on debug build'
    inputs:
      testAssemblyVer2: '${{ parameters.DebugBuildFolder }}\FrameworkUnitTests.dll'

We recently added an empty MSTest project to our solution, which targets Core 6 instead:


  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>

    <IsPackable>false</IsPackable>

    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.8" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.8" />
    <PackageReference Include="coverlet.collector" Version="3.1.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

</Project>

With that project simply present, an error message appears and not a single unit test is discovered. Despite that new MSTest project not being referenced by any other project, and despite VSTest correctly only being set on the same FrameworkUnitTests.dll as before. For every test class xxxTests, the log contains a message of the form:

[MSTest][Discovery][D:\a\1\s\Build\Debug\UnitTests.dll] MSTestAdapter failed to discover tests in class FrameworkUnitTests.xxxTests' of assembly 'D:\a\1\s\Build\Debug\FrameworkUnitTests.dll' because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation..

and at the end there is a message:

No test is available in D:\a\1\s\Build\Debug\FrameworkUnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

The problem does not arise in Visual Studio, I guess because it doesn't need the TestAdapter but runs the tests directly. Notably, no project in the solution referenced the new MSTest project or it's single (empty, default) unit test. As soon as we remove the new Core MSTest project from the solution, all tests get discovered again. Weird!

Now, I was able to discover the potential root of the problem:

The Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll in the build output folder has a different size when the Core MSTest project is present - it's just 59kb, compared to109kb before. I then discovered that because of the Core 6 project, the NuGet reference to MSTest.TestAdapter - which both shows as version 2.2.8 - will actually resolve to different dlls than before. Namely, it uses the dll from mstest.testadapter\2.2.8\build\netstandard1.5 instead of mstest.testadapter\2.2.8\build_common, which targets Framework 4.5

The class TestContextImplementation in Framework 4.5 contains a public DataRow attribute, while the TestContextImplementation in Standard 1.5 does not. The moment the Core project is added to the solution, MSBuild will copy the Standard 1.5 dll into the output directory. But the Framework 4.5 UnitTest.dll was linked against the Framework 4.5 version, where the DataRow attribute IS present. When VSTest loads that library it discovers that the implementation for DataRow is missing and quits. (I don't know why Standard 1.5 doesn't need this implementation, since it also should have to implement https://docs.microsoft.com/de-de/dotnet/api/microsoft.visualstudio.testtools.unittesting.testcontext?view=visualstudiosdk-2022 - but it apparently doesn't?)

I don't currently know how to solve this problem - there seems to be no way to force the 4.8 project to also use the Standard 1.5 version for the MSTest package, thereby linking against the same library that the other project uses, see also: https://github.com/dotnet/sdk/issues/1791 https://github.com/NuGet/Home/issues/7416

I don't know if it is actually well-defined which Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll gets copied to the output folder of a build, is there a way to control this? After all there is a factual conflict between the two libraries - but maybe this isn't discovered since both officially register as MSTest 2.2.8. We consistently see the Standard 1.5 library copied to the output on our Azure DevOps pipeline, so the problem means no tests are discovered for our Framework VSTest above.

One workaround I see right now would be to put the unit tests for Core and Framework into different solutions. It would make sense for us to use the same solution however, since we have projects targetting Framework, newer ones targetting Core and some common projects referenced by both the Framework and Core projects targetting Standard 2.0 for compatibility. In any case, MSTest surely shouldn't just break when both Framework and Core projects use it in a solution.

Another workaround might be be to remove the TestAdapter libraries from the build output folder and run VSTest with a /TestAdapterPath parameter pointing to the correct MSTest version, depending on which project gets tested. But that's rather hacky, especially since I wouldn't know of a nice way to always copy one version of the NuGet package to, say, a folder "FrameworkTestAdapter" and the other to "CoreTestAdapter" so I can reference these in different VSTest tasks.

But maybe there is a way to fix this problem "properly"? For instance, I don't see why it's necessary that the TestContextImplementation exposes public members that simply aren't present in the

Steps to reproduce

Create and build two MSTest projects in the same solution, one targetting Core, the other targetting Framework 4.8

Expected behavior

The Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll that gets copied to the build folder works for running both MSTest projects

Actual behavior

The above dll targeting netstandard1.5 is copied to the build folder. For the Framework MSTest project, running the unit tests from the dll via vstest.console will produce an error: [MSTest][Discovery][D:\a\1\s\Build\Debug\UnitTests.dll] MSTestAdapter failed to discover tests in class FrameworkUnitTests.xxxTests' of assembly 'D:\a\1\s\Build\Debug\FrameworkUnitTests.dll' because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation..

Environment

Operating system: Windows 11
Build version of vstest.console: 17.0.0
Package version of MSTest framework and adapter: 2.2.8
Other installed packages and their versions on the test project: see above

AF250329 commented 2 years ago

I hit this issue too... This is strange that Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll compiled as [assembly: TargetFramework(".NETPortable,Version=v4.5,Profile=Profile259", FrameworkDisplayName = ".NET Portable Subset")] but it always prefer (?) .NET Core version of Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll if one of the projects in solution is .net core based as in description above.... Very strange behavior that depend on how the start-up project was compiled (to full .NET Framework or .NET Core version....) I mean there's vstest.console.exe and testhost.net461.exe in the way before it get to adapter....

feruilob commented 2 years ago

I had a netstandard2.0 project and another multi-target project (net472, netcorapp3.1) to test the netstandard2.0 project and I was hitting this issue. Adding \<AppendTargetFrameworkToOutputPath>true\<\/AppendTargetFrameworkToOutputPath> to all projects helped because now the tests were not getting confused between assembly versions now that each assembly had its own unique path. Hope that helps!

Evangelink commented 2 years ago

@Shanayara @AF250329 @feruilob Are you still facing the issue with 2.2.10? If so, would you be able to provide a reproducer so we can work on a fix?

wangyuan8421 commented 2 years ago

@Evangelink Yesterday I debug this issue a little bit. The issue is still there. Here is the step to reproduce it.

Windows 11, Visual Studio 2022, create a MSTEST project in Net 6.0
It will install ["MSTest.TestAdapter", "MSTest.TestFramework"] version 2.2.8 by default
The unit test will run and test without problem
Then, I upgrade the ["MSTest.TestAdapter", "MSTest.TestFramework"] >= 2.2.10
The Visual Studio test explorer will not find any unit test from dll
Test explorer warning

Evangelink commented 2 years ago

@wangyuan8421 Can you see missing method get_DataRow in the logs? If not, I think this might be a different problem for which I would recommend to open a separate issue.

wangyuan8421 commented 2 years ago

Ah, @Evangelink, I checked the log and it missing get_properties. I will create a new issue then

vinayakmsft commented 2 years ago

Hi @wangyuan8421 / @Evangelink is this issue resolved?

HarmanveerKaur commented 2 years ago

Hi All, I am facing an issue in a release pipeline and this issue looks similar to what i am facing.

VStest task in the release pipeline is not able to find a dll although it is present in the build artefact. Error says- The test source file "D:..........bin\x64\Release\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll " provided was not found.

MSTest Adapter version - 2.1.2 .Net Framework version - 4.7.2

@wangyuan8421 , @Evangelink is this issue still open?

wangyuan8421 commented 2 years ago

@vinayakmsft @HarmanveerKaur this issue is not solve yet. but I have a work around. what I found is another issue. FYI: https://github.com/microsoft/testfx/issues/1256

HarmanveerKaur commented 2 years ago

190446448-71eae19e-ce85-4392-9028-b3fae1369c37

@wangyuan8421 are you talking about this workaround?

Evangelink commented 2 years ago

Can you send us a zip or git repo reproducing the issue so we can test if current main of mstest fixed the issue or if we need to work on a fix?

vinayakmsft commented 2 years ago

@Evangelink Is there a DRI- handle name for issues related to Mstest test adpater/test platform?

HarmanveerKaur commented 2 years ago

@Evangelink I have sent the details on your microsoft email. Kindly take a look.

Evangelink commented 2 years ago

@Evangelink Is there a DRI- handle name for issues related to Mstest test adpater/test platform?

Hi @vinayakmsft. What do you mean by DRI handle name?

vinayakmsft commented 2 years ago

Hi @Evangelink its basically the ICM owner team name. So , can you please let me know what your ICM team name?

Evangelink commented 2 years ago

@vinayakmsft @HarmanveerKaur Sorry for the ping but I don't recall if we have sorted out this issue offline or is there still something to do?

Evangelink commented 1 year ago

I will move forward by closing this issue. Please let me know if there was something else to be done.

bmacombe commented 9 months ago

If it helps some future person running across a similar issue, I resolved it by forcing the MSTestAdapter reference in my project.

<Reference Include="Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter">     
   <HintPath>$(PkgO2S_MSTest_TestAdapter\build\net462\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll</HintPath> 
   <Private>True</Private>
</Reference>

eXpl0it3r commented 1 month ago

This is indeed an issue, if you write all the contents to the same output directory.

Using MSBuild's binary log and MSBuild Structured Log Viewer, you can easily see the double writes, that causes this kind of issue.

StructuredLogViewer_zOAiOCPrtj

If your test project for example runs with .NET Framework 4.8, but references a .NET Standard 2.0 library (e.g. a Source Code Generator project), you end up with having the netfx and netstandard version being written both to the same output directory. Now, if the netstandard version is written last, you end up with the netfx test not being able to find the needed methods, as it can only find the netstandard assembly.

microsoft / testfx