Zero tests ran when using SonarCloud with "sonar.dotnet.excludeTestProjects=true"

[StefH]

When running TUnit tests in Azure DevOps on a Linux image, no tests are found?

Microsoft (R) Code Coverage Command Line Tool (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

SessionId: efba60a0-d999-45b9-861b-f873f497f2a5
  Run tests: '/home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll' [net8.0|x64]
  Failed! - Failed: 0, Passed: 0, Skipped: 0, Total: 0, Duration: 64ms

  Test run summary: Zero tests ran - /home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll (net8.0|x64)
    total: 0
    failed: 0
    succeeded: 0
    skipped: 0
    duration: 123ms
/home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll : error run failed: Tests failed: '/home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/TestResults/WireMock.Net.TUnitTests_net8.0_x64.log' [net8.0|x64] [/home/vsts/work/1/s/test/WireMock.Net.TUnitTests/WireMock.Net.TUnitTests.csproj]
Code coverage results: wiremock-coverage-tunit.xml.

Link to failed DevOps Build: (https://dev.azure.com/stef/WireMock.Net/_build/results?buildId=9212&view=logs&j=3bb91939-0fa9-522e-f055-9541addbc62d&t=dd3b8b64-cf8b-5723-be35-20e4eb86e2a7)

Command to run:

- task: CmdLine@2
    inputs:
      script: |
          dotnet-coverage collect "dotnet test ./test/WireMock.Net.TUnitTests/WireMock.Net.TUnitTests.csproj --configuration Debug --no-build --framework net8.0" -f xml -o "wiremock-coverage-tunit.xml"
    displayName: 'Execute WireMock.Net.Tests with Coverage'

[StefH]

Using version 0.1.807 and the latest.

[thomhurst]

@StefH but it works locally?

[StefH]

Running dotnet test locally on WSL Linux Ubuntu works.

 Run tests: '/mnt/c/dev/GitHub/WireMock.Net/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll' [net8.0|x64]

  ████████╗██╗   ██╗███╗   ██╗██╗████████╗
  ╚══██╔══╝██║   ██║████╗  ██║██║╚══██╔══╝
     ██║   ██║   ██║██╔██╗ ██║██║   ██║
     ██║   ██║   ██║██║╚██╗██║██║   ██║
     ██║   ╚██████╔╝██║ ╚████║██║   ██║
     ╚═╝    ╚═════╝ ╚═╝  ╚═══╝╚═╝   ╚═╝

     v0.1.812.0 | 64-bit | Ubuntu 20.04.6 LTS | linux-x64 | .NET 8.0.8 | Microsoft Testing Platform v1.4.0

  09/24/2024 14:38:48 [Info] : By Stef Heyenrath (https://github.com/WireMock-Net/WireMock.Net)

  09/24/2024 14:38:49 [Info] : Server using .NET 8.0

  Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 979ms

  Test run summary: Passed! - /mnt/c/dev/GitHub/WireMock.Net/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll (net8.0|x64)
    total: 1
    failed: 0
    succeeded: 1
    skipped: 0
    duration: 1s 036ms
  Tests succeeded: '/mnt/c/dev/GitHub/WireMock.Net/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll' [net8.0|x64]

Running with dotnet-coverage collect "dotnet test" -f xml -o "wiremock-coverage-tunit.xml" also works.

[thomhurst]

Hmm... I'll see if I can spin up my own azure pipeline.

In the mean-time, I don't know if it'll have anything useful in it, but could you get access to the log it produced?

/home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/TestResults/WireMock.Net.TUnitTests_net8.0_x64.log

[thomhurst]

Hmm so I've ran the same command on an azure ubuntu pipeline and it's running fine for me annoyingly: https://dev.azure.com/thomhurst/thomhurst/_build/results?buildId=625&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=5caf77c8-9b10-50ef-b5c7-ca89c63e1c86

It's like the source generators aren't running for you, but I can't see any error output or anything.

[StefH]

Note that my test project uses TUnit and the library being tested is using TUnit.Core

[thomhurst]

I've just pushed out a new version: 0.1.817

Could you try upgrading to that and use the --diagnostic cli flag. Then in your bin/TestResults folder in the output directory, you'll have a *.diag file after your test run.

If I could see that it would help determine if it's actually discovering anything.

[StefH]

How to use that flag? When I run this in Windows 11 I get:

PS C:\Dev\GitHub\WireMock.Net\test\WireMock.Net.TUnitTests> dotnet test --diagnostics
MSBUILD : error MSB1001: Unknown switch.
    Full command line: 'C:\Program Files\dotnet\sdk\8.0.400\MSBuild.dll -maxcpucount -verbosity:m -nologo -restore -target:VSTest -nodereuse:false -nologo --diagnostics -property:VSTestArtifactsProcessingMode=collect -property:VSTestSessionCorrelationId=12548_6fc1f26d-e203-4e4f-8a08-045f136eb150 -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,C:\Program Files\dotnet\sdk\8.0.400\dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,C:\Program Files\dotnet\sdk\8.0.400\dotnet.dll'
  Switches appended by response files:
Switch: --diagnostics

For switch syntax, type "MSBuild -help"

[thomhurst]

With dotnet test it needs to go after a -- so that it's parsed as a program flag, not a dotnet flag.

So this should work: dotnet test -- --diagnostic

[StefH]

The WireMock.Net.TUnitTests_net8.0_x64.log shows ��.

The diag file shows:

024-09-24T19:22:51.8314544+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Version: 1.4.0+49599419d351132489cacfed32968dcb4fc1b84a
2024-09-24T19:22:51.8518585+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Logging mode: asynchronous
2024-09-24T19:22:51.8564549+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Logging level: Trace
2024-09-24T19:22:51.8564854+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION CreateBuilderAsync entry time: 19:22:51.792
2024-09-24T19:22:51.8587143+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION PID: 3291
2024-09-24T19:22:51.8588907+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Runtime information: linux-x64 - .NET 8.0.8
2024-09-24T19:22:51.8589106+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Runtime location: /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.8/System.Private.CoreLib.dll
2024-09-24T19:22:51.8589369+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION IsDynamicCodeSupported: True
2024-09-24T19:22:51.8589413+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Test module: /home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.dll
2024-09-24T19:22:51.8593829+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION Command line arguments: '--internal-msbuild-node /tmp/575bacb95d8449fb9988707dbc46af7b/.p --diagnostic'
2024-09-24T19:22:51.8622637+00:00 Microsoft.Testing.Platform.Builder.TestApplication DEBUG Machine info:
Machine name: fv-az408-233
OSVersion: Unix 6.8.0.1014
ProcessorCount: 2
Is64BitOperatingSystem: True
TotalAvailableMemoryBytes(GB): 7

2024-09-24T19:22:51.8623740+00:00 Microsoft.Testing.Platform.Builder.TestApplication INFORMATION TESTINGPLATFORM_DEFAULT_HANG_TIMEOUT: ''
2024-09-24T19:22:51.8848070+00:00 Microsoft.Testing.Platform.Hosts.TestHostBuilder INFORMATION Setting RegisterEnvironmentVariablesConfigurationSource: 'True'
2024-09-24T19:22:51.8924708+00:00 Microsoft.Testing.Platform.Configurations.JsonConfigurationSource+JsonConfigurationProvider INFORMATION Config file '/home/vsts/work/1/s/test/WireMock.Net.TUnitTests/bin/Debug/net8.0/WireMock.Net.TUnitTests.testconfig.json' not found.
2024-09-24T19:22:51.8953669+00:00 Microsoft.Testing.Platform.Hosts.TestHostBuilder INFORMATION Setting PlatformExitProcessOnUnhandledException: 'False', config file: False environment variable:  VSTest mode: False
2024-09-24T19:22:51.9731461+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG TestApplicationOptions.EnableTelemetry: True
2024-09-24T19:22:51.9732049+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG TESTINGPLATFORM_TELEMETRY_OPTOUT environment variable: ''
2024-09-24T19:22:51.9732103+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG DOTNET_CLI_TELEMETRY_OPTOUT environment variable: ''
2024-09-24T19:22:51.9732262+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG TESTINGPLATFORM_NOBANNER environment variable: ''
2024-09-24T19:22:51.9732300+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG DOTNET_NOLOGO environment variable: '1'
2024-09-24T19:22:51.9732319+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG Telemetry is 'ENABLED'
2024-09-24T19:22:51.9733774+00:00 Microsoft.Testing.Platform.Telemetry.TelemetryManager DEBUG Telemetry collector provider: 'Microsoft.Testing.Platform.Telemetry.NopTelemetryService'
2024-09-24T19:22:52.1123796+00:00 Microsoft.Testing.Platform.Hosts.ConsoleTestHost INFORMATION Starting test session 'SessionUid { Value = 97081d3c-b56a-4eab-8258-c13578b7a443 }'
2024-09-24T19:22:52.1193222+00:00 Microsoft.Testing.Platform.OutputDevice.TerminalOutputDevice TRACE DisplayBeforeSessionStartAsync
2024-09-24T19:22:52.1283057+00:00 Microsoft.Testing.Platform.Requests.TestHostTestFrameworkInvoker INFORMATION Test framework UID: 'TUnitExtension' Version: '0.1.817.0' DisplayName: 'TUnit' Description: 'TUnit Framework for Microsoft Testing Platform'
2024-09-24T19:22:52.1449990+00:00 TUnit.Engine.Services.TestsLoader TRACE Found 0 before filtering.
2024-09-24T19:22:52.1453892+00:00 TUnit.Engine.Services.TestFilterService TRACE No test filter found.
2024-09-24T19:22:52.1456110+00:00 TUnit.Engine.Services.TUnitTestDiscoverer TRACE Found 0 tests after filtering.
2024-09-24T19:22:52.1460209+00:00 TUnit.Engine.Services.TUnitTestDiscoverer TRACE 0 tests failed to initialize.
2024-09-24T19:22:52.1831017+00:00 Microsoft.Testing.Platform.Messages.AsynchronousMessageBus TRACE The producer '' (ID: TestHostTestFrameworkInvoker) pushed data:
TestRequestExecutionTimeInfo { DisplayName = TestRequestExecutionTimeInfo, Description = Information about the test execution times., TimingInfo = TimingInfo { StartTime = 09/24/2024 19:22:52 +00:00, EndTime = 09/24/2024 19:22:52 +00:00, Duration = 00:00:00.0486682 } }

2024-09-24T19:22:52.2382359+00:00 Microsoft.Testing.Platform.Hosts.ConsoleTestHost INFORMATION Test session 'SessionUid { Value = 97081d3c-b56a-4eab-8258-c13578b7a443 }' ended with exit code '8' in 00:00:00.1880021

[thomhurst]

Yeah 0 tests are discovered and no test node messages are being sent. Tests are registered via source generation, so if none are registered, your source generator is either failing to execute, or it's just not executing for some reason...

I couldn't see any source generator failure messages in your pipeline.

And I don't know why they wouldn't run at all.

This is stumping me 😕

[StefH]

Maybe because of the --no-build ?

[StefH]

This is my PR btw https://github.com/WireMock-Net/WireMock.Net/pull/1179.

[thomhurst]

But you've done a build in the previous step so all the code should be generated by that.

[StefH]

But you've done a build in the previous step so all the code should be generated by that.

correct

[thomhurst]

@StefH Okay so this is REALLY weird. I forked your repo/branch, and recreated your pipeline on my own azure devops. I couldn't run some bits due to missing marketplace actions and tokens, so I disabled those steps.

And now it's passing with 1 test successfully run!

https://dev.azure.com/thomhurst/thomhurst/_build/results?buildId=656&view=logs&j=3bb91939-0fa9-522e-f055-9541addbc62d&t=554ab44d-33cc-5d6f-6fa1-ce7c2a223e4c

I wonder if sonarscanner is causing it or doing something weird? I don't know what that's doing behind the scenes.

Does it pass for you if you disable that step at all?

[thomhurst]

You can see the last 3 commits I pushed here: https://github.com/thomhurst/WireMock.Net/pull/1

[StefH]

Thank you for analyzing this issue. Tomorrow I'll try disabling sonar and maybe other steps.

I keep you updated.

[SaschaBa]

@StefH I would be interested, whether SonarQube works with TUnit, in case of code coverage and code smells. Would be nice to know it, if your pipeline is running. Thanks.

[thomhurst]

@StefH @SaschaBa I found this issue which is causing Sonar to break source generators: https://github.com/SonarSource/sonar-scanner-msbuild/issues/1469

You're setting the flag that causes the problem so it's most likely that

[StefH]

I can confirm that removing the sonar.dotnet.excludeTestProjects=true does indeed solve my issue and the tests + SonarCloud runs fine.

[thomhurst]

Brilliant. I'll close this then if all is good as this is a sonar scanner bug