[BUG] Testing: MSTest: .NET8 Tests projects that compiles fine are neither visible in Test Explorer, nor runnable

omatrot commented 4 months ago

Describe the Issue

I have a solution with multiple tests projects, but some of them are not visible in Test Explorer. The test methods in the concerned projects are not runnable because there is no green play button. I know that the project needs to compile, and be compiled to have something to run. In my case, all the projects compiles correctly, by only some of them are discovered. Sometimes, I have been able to solve the problem by removing recursively the obj/bin directory of the project and its own project references, then rebuilding the test project. Until now it worked. But not anymore.

The projects have been upgraded in place with the assistant from .NET 4.7.1 to .NET 8.

Steps To Reproduce

I am running VS Code, Insider version, on a M1 Mac. I have absolutely no idea if I would be able to reproduce the problem by creating new solution/projects from scratch. If I add new test projects to my current solution and build them, they are discovered. So this is clearly a problem with some of the project themselves.

I have tried to replace the project content from a non working project with the content of one that works but it doesn't solve the problem.

Expected Behavior

Of course, the Tests should be discovered.

Environment Information

OS: macOS Sonoma 14.3.1
C# Dev Kit v1.5.4 (pre-release) -VS Code Version: 1.88.0-insider (Universal) Commit: 99a19815253d91900be5ec1016e0ecc7cc9a6950 Date: 2024-03-14T05:50:25.208Z Electron: 28.2.6 ElectronBuildId: 27476517 Chromium: 120.0.6099.291 Node.js: 18.18.2 V8: 12.0.267.19-electron.0 OS: Darwin arm64 23.3.0

AbhitejJohn commented 3 months ago

@omatrot : Thanks for letting us know. I'm guessing its not really possible for you to share your solution publicly. Any chance you could share with us the contents of the pane - "Output -> C# Dev Kit - Test Explorer".

Also the contents of one of your test project file that does not work would be useful as well. I'm primarily interested in the package references this project has.

omatrot commented 3 months ago

Sorry for the delay, the problem vanished on my MacBook... But, I have switched to my iMac, no tests are visible. Here is the output of the C# Dev Kit - Test Explorer Output (Some parts are in French, sorry):

Created Test Controller
Utilisation de vstest à partir du SDK dotnet dans [/usr/local/share/dotnet/sdk/8.0.100].
Initialized Test Explorer Server [45656]
Test Store Folder: /Users/oliviermatrot/Library/Application Support/Code - Insiders/User/workspaceStorage/d6565803bf63dc5ce5492de9d48b1bd9/ms-dotnettools.csdevkit
Magasin de données de test ouvert en 0,032 s.
Initialized project system provider.
4 projects added, 0 changed, 0 removed.
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Core.Test/bin/Debug/net8.0/SC.TM.Core.Test.dll [0]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Labs.Test/bin/Debug/net8.0/SC.TM.Labs.Test.dll [0]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Agent.Test/bin/Debug/net8.0/SC.TM.Agent.Test.dll [0]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.ResumeParser.test/bin/Debug/net8.0/SC.TM.ResumeParser.test.dll [0]
0 projects added, 0 changed, 0 removed.

Below is a .csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <SccProjectName>SAK</SccProjectName>
    <SccLocalPath>SAK</SccLocalPath>
    <SccAuxPath>SAK</SccAuxPath>
    <SccProvider>SAK</SccProvider>
  </PropertyGroup>
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <!--<RuntimeIdentifier>osx-arm64</RuntimeIdentifier>-->
  </PropertyGroup>
  <ItemGroup>
    <EmbeddedResource Include="Resources\smiley-green.png" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\SC.TM.Core\SC.TM.Core.csproj" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
    <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" />
    <PackageReference Include="EntityFramework" Version="6.4.4" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.*" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.2" />
    <PackageReference Include="SendGrid.SmtpApi" Version="1.4.6" />
  </ItemGroup>
  <ItemGroup>
   <None Remove="appsettings.json" />
 </ItemGroup>
 <ItemGroup>
   <Content Include="appsettings.json">
     <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
   </Content>
 </ItemGroup>
  <Target Name="CopyAppConfig" AfterTargets="Build" DependsOnTargets="Build">
    <CreateItem Include="$(OutputPath)$(AssemblyName).dll.config">
        <Output TaskParameter="Include" ItemName="FilesToCopy" />
    </CreateItem>
    <Copy SourceFiles="@(FilesToCopy)" DestinationFiles="$(OutputPath)testhost.dll.config" />
  </Target>
</Project>

omatrot commented 3 months ago

By contrast, here is the output from the C# Dev Kit - Test Exploreroutput on my MacBook:

Created Test Controller
Using vstest from dotnet sdk in [/usr/local/share/dotnet/sdk/8.0.201].
Initialized Test Explorer Server [71175]
Test Store Folder: /Users/oliviermatrot/Library/Application Support/Code - Insiders/User/workspaceStorage/8249bb5b498661477643911855b0189d/ms-dotnettools.csdevkit
Test data store opened in 0.055 sec.
Initialized project system provider.
5 projects added, 0 changed, 0 removed.
0 projects added, 0 changed, 0 removed.
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Core.Test/bin/Debug/net8.0/SC.TM.Core.Test.dll [1710423943398]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.SmartTag.Test/bin/Debug/net8.0/SC.TM.SmartTag.Test.dll [1710851659284]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Locality.Test/bin/Debug/net8.0/SC.TM.Locality.Test.dll [1710434169227]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.ResumeParser.test/bin/Debug/net8.0/SC.TM.ResumeParser.test.dll [1710946431558]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Analytics.Test/bin/Debug/net8.0/SC.TM.Analytics.Test.dll [1710799182520]
========== Starting test discovery ==========
154 tests discovered from /Users/oliviermatrot/Documents/SC/Product/SC.TM.Locality.Test/bin/Debug/net8.0/SC.TM.Locality.Test.dll;/Users/oliviermatrot/Documents/SC/Product/SC.TM.Analytics.Test/bin/Debug/net8.0/SC.TM.Analytics.Test.dll;/Users/oliviermatrot/Documents/SC/Product/SC.TM.ResumeParser.test/bin/Debug/net8.0/SC.TM.ResumeParser.test.dll;/Users/oliviermatrot/Documents/SC/Product/SC.TM.Core.Test/bin/Debug/net8.0/SC.TM.Core.Test.dll;/Users/oliviermatrot/Documents/SC/Product/SC.TM.SmartTag.Test/bin/Debug/net8.0/SC.TM.SmartTag.Test.dll in 1669.9793 ms
========== Test discovery finished: 154 Tests found in 1.7 sec ==========

ocallesp commented 3 months ago

Sorry for the delay, the problem vanished on my MacBook... But, I have switched to my iMac, no tests are visible. Here is the output of the C# Dev Kit - Test Explorer Output (Some parts are in French, sorry):

Created Test Controller
Utilisation de vstest à partir du SDK dotnet dans [/usr/local/share/dotnet/sdk/8.0.100].
Initialized Test Explorer Server [45656]
Test Store Folder: /Users/oliviermatrot/Library/Application Support/Code - Insiders/User/workspaceStorage/d6565803bf63dc5ce5492de9d48b1bd9/ms-dotnettools.csdevkit
Magasin de données de test ouvert en 0,032 s.
Initialized project system provider.
4 projects added, 0 changed, 0 removed.
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Core.Test/bin/Debug/net8.0/SC.TM.Core.Test.dll [0]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Labs.Test/bin/Debug/net8.0/SC.TM.Labs.Test.dll [0]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.Agent.Test/bin/Debug/net8.0/SC.TM.Agent.Test.dll [0]
Scheduling discovery: /Users/oliviermatrot/Documents/SC/Product/SC.TM.ResumeParser.test/bin/Debug/net8.0/SC.TM.ResumeParser.test.dll [0]
0 projects added, 0 changed, 0 removed.

Below is a .csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <SccProjectName>SAK</SccProjectName>
    <SccLocalPath>SAK</SccLocalPath>
    <SccAuxPath>SAK</SccAuxPath>
    <SccProvider>SAK</SccProvider>
  </PropertyGroup>
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <!--<RuntimeIdentifier>osx-arm64</RuntimeIdentifier>-->
  </PropertyGroup>
  <ItemGroup>
    <EmbeddedResource Include="Resources\smiley-green.png" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\SC.TM.Core\SC.TM.Core.csproj" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
    <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" />
    <PackageReference Include="EntityFramework" Version="6.4.4" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.*" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.2" />
    <PackageReference Include="SendGrid.SmtpApi" Version="1.4.6" />
  </ItemGroup>
  <ItemGroup>
   <None Remove="appsettings.json" />
 </ItemGroup>
 <ItemGroup>
   <Content Include="appsettings.json">
     <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
   </Content>
 </ItemGroup>
  <Target Name="CopyAppConfig" AfterTargets="Build" DependsOnTargets="Build">
    <CreateItem Include="$(OutputPath)$(AssemblyName).dll.config">
        <Output TaskParameter="Include" ItemName="FilesToCopy" />
    </CreateItem>
    <Copy SourceFiles="@(FilesToCopy)" DestinationFiles="$(OutputPath)testhost.dll.config" />
  </Target>
</Project>

Hi,

Is the same project opened in two different machines, but in one machine it works and in the other doesn't ?

is this a log of a repro? If yes, it looks like 'test discovery' was not executed. This might be the issue.

To have more context, how many projects are in total ? In see "4 projects added, 0 changed, 0 removed.", but I have no idea how many are missing.

For the project having the issue, was the dll generated in the output folder ?

omatrot commented 3 months ago

Is the same project opened in two different machines, but in one machine it works and in the other doesn't ?

Yes it was, the tests that weren't visible in the test pane last friday, are this morning after a VS Code restart, tied to an update, and a solution rebuild.

is this a log of a repro?

Yes

To have more context, how many projects are in total ?

10 test projects, 10 library projects. 1 library has a corresponding unit test project.

For the project having the issue, was the dll generated in the output folder ?

Yes.

github-actions[bot] commented 3 months ago

This issue has been marked as stale after 14 days of inactivity. @abhitejjohn, could you please take a look?

ocallesp commented 3 months ago

I am taking a look at this issue. Will provide updates during the week.

LazerFX commented 3 weeks ago

I had this issue, and ended up dropping back to SDK v7 in order to use the dev kit test explorer. At that point, I started getting Scheduling discovery ... lines appearing in my C# Dev Kit - Test Explorer window. I'll be following along with interest in the results of this investigation.

wjrogers commented 2 weeks ago

I had the same problem this morning working in Ubuntu WSL via the WSL remote extension. Only 3/5 test projects in my solution appeared in Test Explorer. The Output pane had a "Scheduling discovery:" message for all 5, but 2 of them never ran. I tried deleting all build artifacts and restarting vscode, and I tried updating the dotnet SDK, but neither changed the outcome.

Then, I tried updating the NuGet dependencies of my test projects and restarting vscode, and all 5 projects were correctly discovered.

GitHubActionsTestLogger from 2.3.3 to 2.4.1
Microsoft.NET.Test.Sdk from 17.9.0 to 17.10.0
xunit from 2.8.0 to 2.8.1
xunit.runner.visualstudio from 2.8.0 to 2.8.1

Hope this helps track down the problem.

LazerFX commented 2 weeks ago

I've since moved to a different laptop and it's worked flawlessly on exactly the same solution. The main difference is that the laptop is significantly more performant (16GB of ram, and a 12-core CPU rather than a 4 core)... this leads me to suspect this is some form of time-out issue, as it was slightly intermittent in the first place.

microsoft / vscode-dotnettools