dotnet test output quality is degraded in .NET 9 Preview 3 SDK

RikkiGibson commented 1 month ago

Description

See gist https://gist.github.com/RikkiGibson/d9a9cdafa645196c6dd04cd41e3d6ac4. This gist compares output from:

.NET 8 SDK in file dotnet-8-0-101-test-output.log
.NET 9 Preview 3 SDK with msbuild live logging feature enabled in msbuild-live-test-output.log
.NET 9 Preview 3 SDK with live logging disabled in file msbuild-non-live-test-output.log

Note that the specific test failures which are occurring are different in the different log files, but, the characteristics of the logs should be enough to indicate the issues present in them.

Steps to reproduce

Clone Roslyn main branch, modify a test so that it will fail, then run a 'dotnet test' command similar to any of those at the top of the log files. (though you'll have to pick an arbitrary different test class than I was using, if you are filtering the tests, since I was working in a feature branch.)

You can verify the .NET 8 SDK behavior by cloning a release branch like release/dev17.10 and performing the same steps.

Expected behavior

dotnet test output in .NET 9 SDK is at parity with .NET 8 in terms of completeness, brevity, formatting, etc. of the output.

Actual behavior

The key issues in the .NET 9 test output are:

Expected line breaks are not rendered in the test failure error messages, so, the multi-line diagnostic baselines are shown as one long line.
The call stack cuts off and is missing frames.
The path to the test results file is not shown.
In the non-live .NET 9 version, the names of all the succeeding tests are listed, which adds a great deal of noise when reviewing the output.
The erroring test output is labeled with 'error VSTEST1', which seems unrelated to the actual error which is occurring in the user's test code.

Diagnostic logs

Let me know if these are needed in order to diagnose the problem. I'm happy to go back and collect but will go ahead and post this without them for now.

Environment

Windows 11 Pro Version 10.0.22631 Build 22631 .NET 9 SDK Preview 3 Terminal: PowerShell 5.1.22621.2506 in VS Code 1.89.1

AB#2063248

nohwnd commented 1 month ago

Thanks for the repro, I will resume work on terminal logger this week and will have a look.

testplatform-bot commented 1 month ago

✅ Successfully linked to Azure Boards work item(s):

User Story 2063248

nohwnd commented 1 month ago

Preview 4 (or the latest Preview 5), rolled back the changes that sent Passed tests so you won't see then in output anymore, and we've utilized the multiline output that MSBuild added for us. So the output looks much better than before:

We are still not writing the info messages that are coming from XUnit or any other console output, but I will be talking with MSBuild about this soon because it is important for us, e.g. when user wants to just discover tests.

microsoft / vstest