search

coverlet-coverage / coverlet

Cross platform code coverage for .NET
MIT License
2.93k stars 385 forks source link

[BUG] Using MSBuild, the coverage report is not being generated if the file does not exist previously #1587

Closed mikibakaiki closed 4 months ago

mikibakaiki commented 5 months ago

Describe the bug I was trying to use the coverlet.msbuild to make the command dotnet test always run as dotnet test /p:CollectCoverage=true to make life easier for my work colleagues - and myself!

However, this was not working properly. After hours of trying different things, and also running the command dotnet test /p:CollectCoverage=true without any success, i tried creating an empty file called coverage.cobertura.xml and there it was, the report was generated, and it had overwritten the empty file.

The first report with ReportGenerator was empty, because it read the empty file, but in the subsequent runs, it was reading the correct data - I'm currently unsure if it is reading the "freshest" data or just the n-1 run attempt.

To Reproduce Here's my test.csproj setup: 

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

<!-- here, i'm defining the the arguments -->
  <PropertyGroup>
    <CollectCoverage>true</CollectCoverage>
    <CoverletOutput>$(MSBuildProjectDirectory)\coverage\</CoverletOutput>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="coverlet.msbuild" Version="6.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.0" />
    <PackageReference Include="moq" Version="4.20.70" />
    <PackageReference Include="xunit" Version="2.6.5" />
    <PackageReference Include="ReportGenerator" Version="5.2.0" />
     <!-- other packages -->
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\base_location\<the actual code>.csproj" />
  </ItemGroup>

  <Target Name="GenerateCoverageReport" AfterTargets="VSTest">
    <Exec Command="Reportgenerator -reports:$(CoverletOutput)coverage.cobertura.xml -targetdir:$(CoverletOutput)CoverageReport -reporttypes:Html_Dark" />
  </Target>

</Project>

I'm running the command dotnet test from inside the test folder and on the solution folder, and both weren't working until i created the empty file.

Expected behavior I expected to have the coverage.cobertura.xml report being created if the file doesn't exist, and if it does, its content is replaced. Also expected that, after my setup in the test.csproj ,running the command dotnet test would work as an equivalent to dotnet test /p:CollectCoverage=true.

I also tried to run the command dotnet test /p:CollectCoverage=true and the report was not being generated.

Actual behavior I was getting an error: 

PS C:\Users\**\> dotnet test /p:CollectCoverage=true
  Determining projects to restore...
  All projects are up-to-date for restore.
...
Test run for Project.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.7.0-preview-23364-03+bc17bb9693cfc4778ded51aa0ab7f1065433f989 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:    34, Skipped:     0, Total:    34, Duration: 62 ms - Project.dll (net7.0)
  2024-01-15T12:46:16: Arguments
  2024-01-15T12:46:16:  -reports:C:\Users\**\coverage\coverage.cobertura.xml
  2024-01-15T12:46:16:  -targetdir:C:\Users\**\coverage\CoverageReport
  2024-01-15T12:46:16:  -reporttypes:Html_Dark
  2024-01-15T12:46:16: The report file 'C:\Users\**\coverage\coverage.cobertura.xml' is invalid. File does not exist (Full pat
  h: 'C:\Users\**\coverage\coverage.cobertura.xml').
  2024-01-15T12:46:16: No report files specified.
C:\Users\**\Test.csproj(41,5): error MSB3073: The command "Reportgenerator -reports:C:\Users\**\coverage.cobertura.xml -targetdir:C:\Users\**\coverage\CoverageReport -repor 
ttypes:Html_Dark" exited with code 1.

Both paths were correct and were equal. The problem was that the report file was not found.

Configuration (please complete the following information): Please provide more information on your .NET configuration:

Additional context Add any other context about the problem here.

:exclamation: Please also read Known Issues

mikibakaiki commented 5 months ago

I see you added the waiting for customer tag. What else can i provide?

Bertk commented 5 months ago

Please use the latest preview from coverlet nightly build and check whether this is still an issue.

Bertk commented 4 months ago

Could you please execute the test again and use -bl. The binary log file will show the value for property $(CoverletOutput).

The documentation uses forward slash which definitely works e.g.

dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/'

I use the command line and the tried some variations:

status msbuild parameter
:heavy_check_mark: /p:CoverletOutput="C:/GitHub/coverlet/artifacts/Reports/coverlet.core/"
:heavy_check_mark: /p:CoverletOutput="C:\\GitHub\\coverlet\\artifacts\\Reports\\coverlet.core\\"
:heavy_check_mark: /p:CoverletOutput="C:\GitHub\coverlet\artifacts\Reports\coverlet.core\\"
:x: /p:CoverletOutput="C:\GitHub\coverlet\artifacts\Reports\coverlet.core\"

Issue cannot be reproduced with latest preview version.

mikibakaiki commented 4 months ago

I can't seem to use the version you mention.

I see that you're using the CLI commands, and they seem to work. Could you test including the 

  <PropertyGroup>
    <CollectCoverage>true</CollectCoverage>
    <CoverletOutput>$(MSBuildProjectDirectory)\coverage\</CoverletOutput>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
  </PropertyGroup>

and check if it works too? :)

Bertk commented 4 months ago

Issue cannot be reproduced with latest preview version.

@mikibakaiki I tested CoverletOutput property in C# project file already.

<CoverletOutput>C:/GitHub/coverlet/artifacts/Reports/coverlet.collector/</CoverletOutput

Bertk commented 4 months ago

coverlet V6.0.1 is released.