Azure Pipelines no longer shows Tests UI with new SDK

[craigbrown]

I have updated my projects to use the new MSTest SDK:

- <Project Sdk="Microsoft.NET.Sdk">
+ <Project Sdk="MSTest.Sdk/3.4.1">
...

and I have the following task in my Azure Pipeline (I know the Azure tasks are a different repo, but bear with me):

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/**.sln'
    arguments: '--configuration Release'

which seems to translate to this command when it's actually run:

"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\MySolution.sln --logger trx --results-directory D:\a\_temp --configuration Release

This used to result in a "Tests" tab appearing on the pipeline's run page with a nice UI of all the tests that were run and whether they passed or not. After the change to the MSTest.Sdk, this no longer appears and I also get this warning, which I think is related:

##[warning]No test result files were found.

I have searched the issues in this repo for any clue as to how to fix this. I've seen some mention of the trx part possibly being the culprit, so I followed the suggestion to add this to my csproj:

<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="1.0.0" />

but this doesn't seem to make a difference.

Could you point me in the right direction to get this working? Or is it something that the Azure Pipelines team will need to fix on their end? Many thanks.

[Evangelink]

Hi @craigbrown,

First of all, sorry that the upgrade experience wasn't good for you. Let me give you some details about the change you did and what is missing.

Context

By default, dotnet test is calling vstest.console.exe which was the main platform for running tests until we introduced MSTest runner. The 2 platforms/runner are fundamentally different and uses different command line arguments. We then introduced MSTest SDK with the main goal to help simplify boilerplate configuration for our users and promote the new platform. As part of using MSTest.Sdk you are switching the default platform from vstest.console to MSTest runner (as explained here https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-sdk#select-the-runner) and we are also opting in switching dotnet test from calling vstest.console to be calling MSTest runner (this part is not well documented, I'll work on improving that), it's equivalent to be setting up https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#dotnet-test---microsofttestingplatform-mode.

When you have moved from the regular SDK to MSTest.Sdk you are in one of the following 2 scenarios:

1. You were simply using MSTest (either through MSTest metapackage or through MSTest.TestAdapter and MSTest.TestFramework) but you didn't opt-in to the runner (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-intro#enable-mstest-runner-in-a-mstest-project).

2. You were already using MSTest runner but you were not using the runner with dotnet test.

Answer to your question

Since you are using MSTest.Sdk you are already having the required extra packages added through the default extensions profile (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-sdk#mstest-runner-profile) so you only need to pass the argument required to activate the TRX (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-extensions-test-reports#visual-studio-test-reports).

So your yaml should be:

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/**.sln'
    arguments: '--configuration Release -- --report-trx'

Note the -- in front of --report-trx to let the SDK know the arguments after that separator are not for MSBuild but need to be flown down to the platform.

Please let me know if you have any question left.

[craigbrown]

Hi @Evangelink, thank you for your quick and thorough response! I tried your suggestion and unfortunately it doesn't fix the problem for me. I tried running the same command on my machine:

dotnet test --configuration Release -- --report-trx

It seems to run the tests correctly (the command line shows the correct number of passes and fails), but I get the same issue with no trx being output (I am looking in the TestResults folder based on the last link above).

[Evangelink]

Absolutely sorry I got confused with what is currently implemented (many phases to handle got me confused)... As explained in here https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#dotnet-test---microsofttestingplatform-mode, you will need to use -p:TestingPlatformCommandLineArguments="--report-trx".

So

dotnet test --configuration Release -p:TestingPlatformCommandLineArguments="--report-trx"

You can also use -p:TestingPlatformCaptureOutput=false in addition to the previous command to get a more detailed output.

[craigbrown]

Thanks @Evangelink! That does now produce the trx file when I run the command on my machine. Unfortunately it doesn't get picked up by Azure DevOps, but I assume this is something on their side, possibly because they're looking for the trx file in the wrong directory. I will open a separate issue in that repository.

[Evangelink]

@craigbrown I'll try to make some tests on AzDO to see whats going-on. Thanks for linking the issue on their side here, it's helping a lot with triaging and syncing between teams <3

If you have some time, would you mind looking at this doc change PR https://github.com/dotnet/docs/pull/41658 and let me know if that seems to be providing enough info? It's sometimes hard as author to find the right level of details for our users.

[craigbrown]

@Evangelink sure! That looks good. I left a comment with the only change I think is needed to your example. Thanks again for the quick response to this.