Closed xantari closed 5 months ago
Ok there are a few thing off here. First of all coverlet.collector
doesn't have a summary output on the console. This here is definitely not from our collector.
+-------------------------------+-------+--------+--------+
| Module | Line | Branch | Method |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Api.Client | 100% | 100% | 100% |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Services | 3.59% | 1.66% | 26.98% |
+-------------------------------+-------+--------+--------+
+---------+--------+--------+--------+
| | Line | Branch | Method |
+---------+--------+--------+--------+
| Total | 3.59% | 1.66% | 26.98% |
+---------+--------+--------+--------+
| Average | 51.79% | 50.83% | 63.49% |
+---------+--------+--------+--------+
I see that you are additionally referencing coverlet.msbuild
. So it could be from here. This would also explain why the parameter /p:Exclude="[*]GalaxyDigital.Api.*"
somehow has an effect. On the other hand you are calling dotnet test
with a specified collector which doesn't really match to that theory.
What else is a bit weird is that you have specified two different collectors in your .runsettings
file. Coverlet with the friendlyName="XPlat Code Coverage" and the internal one from vstest with the friendlyName="Code Coverage". In the result I then see two coverage files coverage.cobertura.xml
and test_2024-01-31.13_26_01.coverage
. This format .coverage
is not supported by coverlet at all so this can't be created by us (link).
Not sure why but it doesn't seem that this report is from coverlet.
I just tried it out with a simple repro with the following coverlet.runsettings
file:
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
<Configuration>
<Format>cobertura</Format>
<Exclude>[*]GalaxyDigital.Api.*</Exclude>
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
and calling the collector like this:
dotnet test --collect:"XPlat Code Coverage" --settings coverlet.runsettings
and everything works as expected. Please also remove the coverlet.msbuild
reference if you try this again. I hope this helps a bit to find the issue. If you could provide a simple repro I could analyse this more in detail.
So I think my issue is that the MSBuild settings in my csproj was the only way I was getting a coverage.cobertura.xml output to a known path (TestResults/coverage.coburtura.xml). Where as removing the msbuild package and the associated .csproj modifications would only output it to TestResults/{some random guid}/coverage.cobertura.xml
The extra output you were seeing in the output with the coverage was actually the reportgenerator nuget that consumes ccode coverage results (also an msbuild task).
Here is my .csproj:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<Platforms>AnyCPU;x64</Platforms>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="coverlet.collector" Version="6.0.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="coverlet.msbuild" Version="6.0.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="IBM.EntityFrameworkCore" Version="6.0.0.300" />
<PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="6.0.26" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="Moq" Version="4.20.70" />
<PackageReference Include="NUnit" Version="4.0.1" />
<PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
<PackageReference Include="ReportGenerator" Version="5.2.0" />
</ItemGroup>
<PropertyGroup>
<RunSettingsFilePath>$(MSBuildProjectDirectory)\CodeCoverage.runsettings</RunSettingsFilePath>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\GalaxyDigital.Services\GalaxyDigital.Services.csproj" />
</ItemGroup>
<ItemGroup>
<Folder Include="TestResults\" />
</ItemGroup>
<!--https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.msbuild.tasks/coverlet.msbuild.props-->
<PropertyGroup>
<CollectCoverage>true</CollectCoverage>
<CoverletOutput>TestResults\</CoverletOutput>
<CoverletOutputFormat>cobertura</CoverletOutputFormat>
<IncludeTestAssembly>false</IncludeTestAssembly>
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
<Exclude>[*]GalaxyDigital.Console.*,[*]GalaxyDigital.Api.*</Exclude>
</PropertyGroup>
<Target Name="GenerateHtmlCoverageReport" AfterTargets="GenerateCoverageResultAfterTest">
<ReportGenerator ReportFiles="@(CoverletReport)" TargetDirectory="TestResults\html-coverage-report" ReportTypes="HtmlInline" />
</Target>
</Project>
When running this command:
dotnet test --settings CodeCoverage.runsettings --no-build --collect:"XPlat Code Coverage"
I will get a TestResults/coverage.cobertura.xml (a deterministic file output location) so that the reportgenerator can find it.
This file appears to be generated by coverlet.msbuild and using all the settings in the PropertyGroup listed above from my .csproj file. This is then passed to @(CoverletReport) for the report generator. I was able to get the namespaces excluded properly using the above .csproj settings as well.
However I am reading that I shouldn't really be using both coverlet.collector and coverlet.msbuild together.
So I tried to remove coverlet.msbuild, and removed all of these properties from the .csproj:
<PropertyGroup>
<CollectCoverage>true</CollectCoverage>
<CoverletOutput>TestResults\</CoverletOutput>
<CoverletOutputFormat>cobertura</CoverletOutputFormat>
<IncludeTestAssembly>false</IncludeTestAssembly>
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
<Exclude>[*]GalaxyDigital.Console.*,[*]GalaxyDigital.Api.*</Exclude>
</PropertyGroup>
Now, I can no longer run the same command:
dotnet test --settings CodeCoverage.runsettings --no-build --collect:"XPlat Code Coverage"
It will error with:
dotnet : The test source file "C:\TFS\GalaxyDigital\GalaxyDigital.Services.Tests\bin\Debug\net6.0\GalaxyDigital.Services.Tests.dll" provided was not found.
So now I modify it slightly, because it isn't picking up that this is an x64 build:
dotnet test ./bin/x64/Debug/net6.0/GalaxyDigital.Services.Tests.dll --no-build --collect:"XPlat Code Coverage" --settings CodeCoverage.runsettings
Then I get this:
Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'
Data collection : Unable to find a datacollector with friendly name 'Code Coverage'.
Data collection : Could not find data collector 'Code Coverage'
I guess it is because it can't find the nuget path according to this: https://github.com/coverlet-coverage/coverlet/issues/1546
Each dev has their own nuget path, so now i'm stuck because I won't be able to share the project around with different devs due to their nuget location differences. So back to this command which does a build but puts it into /bin/Debug/net6.0 and disobeys that this is a 64-bit only application:
dotnet test --collect:"XPlat Code Coverage" --settings CodeCoverage.runsettings
This works now as it now output another compilation to the wrong bin folder (no x64 in the path).
But then the problem of the random guid paths crops up again such that the report generator cannot determine where to find the coverage.cobertura.xml file.
C:\TFS\GalaxyDigital\GalaxyDigital.Services.Tests\TestResults\28a9b03b-eaa8-4859-b0e9-6470a6aa8e2a\coverage.cobertura.xml
So, back to the beginning, the coverlet.msbuild route seems to be the only way to generate a deterministic file path for the coverage.cobertura.xml file.
I settled on removing all msbuild stuff from .csproj. I created a separate .bat file that users will have to run separately from a command prompt as I couldn't figure out how to get deterministic file names output (visual studio wants to continue to create random guids in the path). And coverlet not outputting itself if it is anything other then an AnyCPU build (x64 specific builds do not output the coverlet dlls for some reason and then give you the missing XPlat Code Coverage data collector errors).
Maybe I am missing something obvious to get the other methods to work...
REM "Removing previous coburtura.xml files so it doesn't get confused which one to process"
rd /s /q "TestResults\"
dotnet test --collect:"XPlat Code Coverage" --settings CodeCoverage.RunSettings
%userprofile%\.nuget\packages\reportgenerator\5.2.0\tools\net8.0\ReportGenerator.exe "-reports:TestResults\*\coverage.cobertura.xml" "-targetdir:TestResults\CoverageReport" --reporttypes:HtmlInline
start TestResults/CoverageReport/index.html
OK I think now it is a bit more clear to me what you want to do. Integrating coverlet
and reportgenerator
into Visual Studio. I admit I've never tried that out. Most users only calculate coverage at some stage in their pipeline.
But there is a Visual Studio extension that uses coverlet
to do exactly this: https://github.com/the-dext/RunCoverletReport. Maybe you can get some information from there how to do this.
When I use coverlet
locally, I'm also using a script pretty similar to what you have now.
Yeah, I basically wanted to continually see how I was doing in regards to branch and code coverage as I was writing unit tests. I sort of got it working iwth the coverlet msbuild stuff, and the reportgenerator msbuild stuff, but it wasn't clear to me that when you go that route the runsettings seem to be ignored and you need to put those settings in the .csproj file instead of a runsettings file.
Closing this. Thanks for your help!
Describe the bug
Namespace / assembly exclusions are not being obeyed from the .runsettings file.
To Reproduce
Example .runsettings:
When running the following "GalaxyDigital.Api.Client" is not excluded:
You get the following output:
Notice how GalaxyDigital.Api.Client was not excluded, even though it was excluded in the runsettings file.
If you change the command line to the following:
Then you get this output:
This properly excluded the namespace "GalaxyDigital.Api.Client" but required the extra command line switch that should have been coming from the runsettings and not require the command line switch.
Expected behavior
.runsetting file excludes should be obeyed when passed into the dotnet command.
Actual behavior
dotnet command requires the /p:Exclude parameter to actually exclude the assembly/namespaces.
Configuration (please complete the following information): Please provide more information on your .NET configuration:
Additional context Add any other context about the problem here.