search

microsoft / vstest

Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
MIT License
869 stars 316 forks source link

Unsuppressable overwrite warning when using <VSTestLogger> MSBuild property which confuses users #5132

Open MattKotsenas opened 2 days ago

MattKotsenas commented 2 days ago

Description

In order to have the developer's local build and CI build as closely aligned as possible, I set the <VSTestLogger> property in a Directory.Build.targets file like this

<Project>
  <PropertyGroup Condition=" '$(IsTestProject)' == 'true' ">
    <!-- Enable test logging to TRX files and place them in the artifacts directory -->
    <VSTestLogger>trx%3bLogFileName=$(MSBuildProjectName).trx</VSTestLogger>
  </PropertyGroup>
</Project>

In CI this is fine because build and test generally only happens once. However, when running locally, subsequent test runs show this warning:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
WARNING: Overwriting results file: C:\path\to/TestResults/net8.0\Moq.Analyzers.Test.trx
Results File: C:\path\to/TestResults/net8.0\Moq.Analyzers.Test.trx

In this scenario, overwriting the test results file is expected. The warning is raised @ https://github.com/microsoft/vstest/blob/a1f5a6500b8cfefa81adbb652a84ad0ba884c140/src/Microsoft.TestPlatform.Extensions.TrxLogger/TrxLogger.cs#L452-L456

In this particular case the warning is benign, however:

  1. There doesn't appear to be a way to suppress it
  2. Even if I could, I'd prefer to not have a warning in this case, since overwriting is expected
  3. I'd prefer not to teach developers to ignore warnings
  4. I'd like to recommend this type of setup as a best practice for reproducible-builds, however the poor developer experience currently prevents that

Steps to reproduce

This should be reproducible just from the snippets provided as context, but for a full example:

  1. Clone https://github.com/rjmurillo/moq.analyzers
  2. Checkout 089a203444a06ecbf18415a21a24b2a943aede62 in case the repo changes
  3. dotnet test
  4. dotnet test

Expected behavior

Actual behavior

Environment

vstest.console.exe 17.0.34804.30 dotnet 8.0.300 Windows 23H2 22635.3790