coverlet-coverage / coverlet

Cross platform code coverage for .NET
MIT License
2.99k stars 386 forks source link

Allow to set root directory for ExcludeAssembliesWithoutSources heuristic or other way to exclude internal nuget dependencies on shared CI nodes. #1668

Open EraYaN opened 4 months ago

EraYaN commented 4 months ago

Currently we are running coverlet in self hosted CI runners and these runners cache other builds as well. This causes coverlet to go up the directory tree to find a bunch of old builds that match some of the assemblies. I'd like to be able to pass the current root so any file outside of that will just be ignored.

We currently run the tests like so:

- task: DotNetCoreCLI@2
  displayName: "Run Tests"
  condition: and(succeeded(), ${{ parameters.runTests }})
  env: ${{ parameters.testEnv }}
  inputs:
    command: "test"
    projects: ${{ parameters.testProjects }}
    arguments: '--configuration Release --collect:"XPlat Code Coverage" --settings "${{ parameters.testSettingsFile }}" --verbosity minimal'
    publishTestResults: true
    testRunTitle: $(Agent.JobName)
    workingDirectory: "$(Build.SourcesDirectory)"

with runsettings liek this:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>
          <Exclude>[coverlet.*.Testss?]*,[*]Coverlet.Core*,[*]Moq*</Exclude> <!-- [Assembly-Filter]Type-Filter -->
          <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
          <SingleHit>false</SingleHit>
          <UseSourceLink>false</UseSourceLink>
          <IncludeTestAssembly>false</IncludeTestAssembly>
          <ExcludeAssembliesWithoutSources>MissingAny</ExcludeAssembliesWithoutSources>
          <InstrumentModulesWithoutLocalSources>false</InstrumentModulesWithoutLocalSources>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

I'd like to inject the $(Build.SourcesDirectory) as a root so everything outside that is recognized as a "Third-Party" assembly. Alternatively, it'd be sweet if this worked by reading the csproj files and excluding all <packagereference> items.

Or maybe parse a .deps.json file and exclude everything that has .libraries[].type=='package'.

The main issue is that we have some very large projects that are dependencies and are tested separately and the report generation for the thousands of types takes forever.

EDIT: I potentially overlooked IncludeDirectories/IncludeDirectory, oops, I'll try running a quick test suite.
UPDATE: -p:IncludeDirectory="$(Build.SourcesDirectory)" does not seem to work to filter out anything outside of this directory.

Bertk commented 3 months ago

Please see (answer) https://github.com/coverlet-coverage/coverlet/discussions/1498

EraYaN commented 3 months ago

That does not seem to properly exclude the files. I was not including those projects in the first place, they live somewhere on the accesible filesystem tree near the root, far from the actual source and working directory. It just so happens that Nuget packages that are installed have those exact paths in their metadata and so coverlet goes and tries to find them, making the heuristics for exclusion fail. Include directory does not seem to work as a negative (exclude everything outside of these directories) when it is specified.

Bertk commented 3 months ago

Microsoft introduced simplified output paths with .NET 8.0. coverlet uses this feature already. This feature might be helpful for your scenario and I think using $(Build.SourcesDirectory) is not an appropriate solution.

Typically the project binaries are separated and coverlet will uses the DLLs within the output folder. Did you customized the build output folder by yourself?