search

coverlet-coverage / coverlet

Cross platform code coverage for .NET
MIT License
2.93k stars 385 forks source link

Question about code coverage of test assemblies #1571

Closed sungam3r closed 5 months ago

sungam3r commented 6 months ago

Hi. I know that coverlet excludes test assemblies by default thanks to default value for IncludeTestAssembly setting. Nevertheless I've noticed that in some test reports for projects in our codebase there are still test assemblies like Company.CoolFeature.UnitTests. After investigation I found that this test assembly is referenced by another test assembly from repo. Why? Well, for example, because Company.CoolFeature.UnitTests shares some test models to use across test assemblies. So my questions are:

  1. Is this behavior expected or a some sort of bug?
  2. Does IncludeTestAssembly work only for "root" test assemblies and not for referenced ones?
  3. What is the recommended way to 100% exclude test assemblies from our code coverage?

Thanks.

Bertk commented 5 months ago

Hi, there are some limitations e.q. generated code for test projects.

You can use /p:ExcludeByAttribute="GeneratedCodeAttribute" and reduce the noise. I use reportgenerator and generate coverage reports. This can be used to tweak the configuration for the coverage reports.

see also coverlet CI:

<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>

sungam3r commented 5 months ago

I have nothing to do with GeneratedCodeAttribute. One test assembly references other test assembly - that is all.

Bertk commented 5 months ago

Please provide more details. In the past we had similar issues and some were created by Microsoft.Net.Test.Sdk code generation. You should also check the filter option see ReportGenerator Command Line Parameter

sungam3r commented 5 months ago

I know about report generator and use it. The question is about coverlet behaviour, not reportgenerator. What details I can provide? I already said that situation is very simple - when one test project become reference other test project then I see coverage for referenced assembly even when it was excluded by filter like [*Tests]*. If I remove reference then filter works as expected.

Bertk commented 5 months ago

Sorry, but could you please explain the target element for the filter e.g. file names will not work.

example exclude filter used in coverlet CI

/p:Exclude="[coverlet.core.tests.samples.netstandard]*%2c[coverlet.tests.xunit.extensions]*%2c[coverlet.tests.projectsample]*%2c[testgen_]*"

daveMueller commented 5 months ago

@sungam3r I think this comment here explains it (https://github.com/coverlet-coverage/coverlet/issues/1503#issuecomment-1783945038).

  1. Is this behavior expected or a some sort of bug?

Yes it is the expected behavior.

  1. Does IncludeTestAssembly work only for "root" test assemblies and not for referenced ones?

Yes exactly. It only works for the test assembly being executed. If the root-(test)-assembly references other assemblies we can't really distinguish whether it is a test assembly or not.

  1. What is the recommended way to 100% exclude test assemblies from our code coverage?

It may be best for your scenario to exclude the assembly with the test models by using filters. The other option would be to really separate the test assemblies in a way that they don't reference each other.

Have I been able to clarify all questions?

sungam3r commented 5 months ago

@daveMueller Thank you, this is exactly what I expected to hear. I had to make sure that coverlet works just like that.