"no tests discovered" in xUnit test project

[kevinbarabash]

Describe the bug

I added a new xUnit test project using the + button in the F# solution explorer. When I did that the project did show up until I restarted VSCode which was a little strange, but no big deal. Once it did recognize the project it said it wasn't able to discover any tests and to check if dotnet test was able to run. I ran dotnet test from the project directory and it ran the one example test correctly.

Steps to reproduce

Add a new xUnit test project using the F# solution explorer in VSCode Restart VSCode if necessary to have the new test project show up See that Ionide shows a warning popup saying that there are "no discovered tests" for the new project

Link to sample reproduction

n/a

Expected behaviour

It should be able to recognize tests in a new xUnit test project.

Screenshots

Machine info

• OS: Mac
• .NET SDK version: 7.0.402
• Ionide version: 7.15.2

Additional context

n/a

[baronfel]

cc @farlee2121 - do you have any tips we could point folks at when the test explorer can't locate their tests? anything you saw frequently when you were developing the feature?

[farlee2121]

Running with dotnet test is a frequent diagnostic check I use if tests don't run. Often there ends up being some kind of error.

While less frequent, another indicator was checking the Ionide solution explorer to make sure the project shows up. More than once I'd forgotten to add the project to the solution.

I noticed when following these reproduction steps that the new project didn't show in the Ionide solution explorer after I used the explorer's add button. This was because my workspace is based on the solution file, and the project wasn't added to the solution yet. Everything (including tests) worked as expected once I added the project to the solution.

[farlee2121]

Ah. Another check I use is the solution explorer's package reference view. I use that to make sure Ionide thinks the project references expected libraries like Microsoft.TestPlatform.TestHost or Microsoft.NET.Test.Sdk and various testing libraries

[baronfel]

Oh that makes perfect sense - great sanity checking there.

[farlee2121]

I forgot an important diagnostic tool. There are also various helpful logs in Output > F# - Test Adapter (select it in the dropdown).

The log starting with Refresh - Test Projects lists the projects the explorer detected as test projects.

If a test project actually runs, you can see the full command used. The log will look something like

Running `dotnet test "path-to-proj.fsproj" --framework:"net6.0" --logger:"trx;LogFileName=path-to-trx.trx" --noLogo --no-build`

[Martin521]

Possibly related to #1960. Do you have test names that contain non-letter characters?

[laurentpayot]

I’m experiencing the same issue in VSCode v1.85.0 on Ubuntu 23.10. Tests files are visible in the solution explorer, tests appear in the test explorer panel, and dotnet test runs the tests all right. But at VSCode startup I get the exact same Ionide popup about "no tests discovered". You can reproduce this here: https://github.com/laurentpayot/fsharp-fable-elmish-example

I tried to fix it with https://github.com/xunit/xunit/issues/1521#issuecomment-337556655, without success. Any idea?

[farlee2121]

Have you tried any of the diagnostic steps outlined above?

[laurentpayot]

When I wrote my first comment my repo had a single project for both the app and the tests. I rewrote it in a more "best practice" way to use a project for the app and a project for the tests (but without solution file). Now Ionide warns me that no tests were discovered in my two projects :neutral_face:

@farlee2121 running dotnet test "app.fsproj" --framework:"net8.0" --logger:"trx;LogFileName=test-log.trx" --noLogo --no-build produced nothing on my screen nor on my repo directory, as far as I can tell.

After I open VSCode, the "F# - Test Adapter" panel shows the following log (note the two contradictory last entries):

[14:12:33 DEBUG] [TestExplorer] Extension Storage /home/laurent/.config/Code/User/workspaceStorage/6266b4c3c2199f6f371e31cf163bdc6d/Ionide.Ionide-fsharp
[14:12:55 DEBUG] [TestExplorer] Refresh - Test Projects [
  '/home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj',
  '/home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj'
]
[14:12:55 INFO ] [TestExplorer] Building 2 test projects
[14:12:55 INFO ] [TestExplorer] Building /home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj
[14:12:57 INFO ] [TestExplorer] Building /home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj
[14:12:59 INFO ] [TestExplorer] Discovering tests for /home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj
[14:12:59 DEBUG] [TestExplorer] Running `dotnet test "/home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj" --framework:"net8.0" --noLogo --list-tests --no-build`
[14:12:59 INFO ] [TestExplorer] Discovering tests for /home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj
[14:12:59 DEBUG] [TestExplorer] Running `dotnet test "/home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj" --framework:"net8.0" --noLogo --list-tests --no-build`
[14:13:00 INFO ] [TestExplorer] Discovered 4 tests
[14:13:00 WARN ] [TestExplorer] No tests discovered for the following projects. Make sure your tests can be run with `dotnet test` 
 /home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj

Also note that the erroneous Ionide warning appears every time I click on "Refresh Tests" icon of the VSCode test explorer panel.

When clicking on the "Run Tests" icon of the VSCode test explorer panel, the following entries are added to the log:

[14:13:48 DEBUG] [TestExplorer] TestRunRequest Oi {
  include: undefined,
  exclude: [],
  profile: E {
    controllerId: 'fsharp-test-controller',
    profileId: -1169225476,
    d: 'Run F# Tests',
    kind: 1,
    runHandler: [Function: c],
    f: true,
    _tag: undefined,
    g: false
  },
  continuous: false
}
[14:13:57 DEBUG] [TestExplorer] Running `dotnet test "/home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj" --framework:"net8.0" --logger:"trx;LogFileName=/home/laurent/.config/Code/User/workspaceStorage/6266b4c3c2199f6f371e31cf163bdc6d/Ionide.Ionide-fsharp/TestResults/app.trx" --noLogo --no-build`
[14:13:57 DEBUG] [TestExplorer] Running `dotnet test "/home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj" --framework:"net8.0" --logger:"trx;LogFileName=/home/laurent/.config/Code/User/workspaceStorage/6266b4c3c2199f6f371e31cf163bdc6d/Ionide.Ionide-fsharp/TestResults/tests.trx" --noLogo --no-build`
[14:13:58 DEBUG] [TestExplorer] Test run exitCode -  - 
[14:13:58 DEBUG] [TestExplorer] Test run exitCode - Test run for /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/tests.dll (.NETCoreApp,Version=v8.0)

Microsoft (R) Test Execution Command Line Tool Version 17.8.0 (x64)

Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

WARNING: Overwriting results file: /home/laurent/.config/Code/User/workspaceStorage/6266b4c3c2199f6f371e31cf163bdc6d/Ionide.Ionide-fsharp/TestResults/tests.trx

Results File: /home/laurent/.config/Code/User/workspaceStorage/6266b4c3c2199f6f371e31cf163bdc6d/Ionide.Ionide-fsharp/TestResults/tests.trx

Passed!  - Failed:     0, Passed:     3, Skipped:     0, Total:     3, Duration: 3 ms
 - tests.dll (net8.0)
 - 

But just like @kevinbarabash I have no issue with running tests, just with the erroneous Ionide message (especially annoying for an example repo).

Thanks for your help if you find something.

[Martin521]

Can you check the "Ionide: MSBuild" output window? In my case I have a build error in one of the projects, which I see in that window.

[laurentpayot]

Sure. At VSCode startup, on the "Ionide: MSBuild" output window, I get:

[15:26:14 INFO ] [msbuild] invoking msbuild from /usr/bin/dotnet on /home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj for target Build
MSBuild version 17.8.3+195e7f5a3 for .NET
  app -> /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/app.dll
[15:26:17 INFO ] [msbuild] invoking msbuild from /usr/bin/dotnet on /home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj for target Build
MSBuild version 17.8.3+195e7f5a3 for .NET
  app -> /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/app.dll
  tests -> /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/tests.dll

When running tests, the following lines are added:

[15:27:45 INFO ] [msbuild] invoking msbuild from /usr/bin/dotnet on /home/laurent/projects/FS/fsharp-fable-elmish-example/app.fsproj for target Build
MSBuild version 17.8.3+195e7f5a3 for .NET
  app -> /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/app.dll
[15:27:49 INFO ] [msbuild] invoking msbuild from /usr/bin/dotnet on /home/laurent/projects/FS/fsharp-fable-elmish-example/tests.fsproj for target Build
MSBuild version 17.8.3+195e7f5a3 for .NET
  app -> /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/app.dll
  tests -> /home/laurent/projects/FS/fsharp-fable-elmish-example/bin/Debug/net8.0/tests.dll

[laurentpayot]

In my case I have a build error in one of the projects, which I see in that window.

Could you tell me what build error you get?

[Martin521]

Could you tell me what build error you get?

Mine was a real error message of MsBuild (I don't have it at hand any more). Your case seems to be different, build seems ok.

Actually, your output above seems to be completely ok to me. If you have tests only in one of your two projects, is it not ok to have that error message?

[farlee2121]

Those logs are helpful. The error message is behaving as expected. Both your app and test projects are referencing key test packages, so it looks like both of them are test projects. But, the app project contains no tests and thus looks like a test project with no tests, which is generally a sign something is off.

[laurentpayot]

Oh I didn’t read the logs I copy-pasted. You are right, the list of projects without tests contains only one element: the app without tests. That’s an appropriate behavior. I don’t know why but I can assure you I had the no test AND the test project appearing in the list :face_with_spiral_eyes: I cannot reproduce the two projects warning right now. Weird.

By the way, I created #1970 for the MSBuild project property group element <IsTestProject> (generated by xUnit for test projects) to be taken into account by Ionide when its value is false.