Multiple test projects - Githubissues

Foshkey commented 1 year ago

Describe the bug First of all, awesome extension. I'm looking for something similar to VS Live Testing and this might be it 😄 Also, apologies if I'm missing something obvious, I tried searching for this issue.

For dotnet, I managed to get this working when I add this package to just one test project:

    <PackageReference Include="coverlet.msbuild" Version="3.2.0">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>

But as soon as I add it to all of my test projects within the same workspace, all covered lines indicate no code coverage within all or all but one project. The lcov.info files are generated within their respective project directories, and I confirmed that they are all different.

To Reproduce Steps to reproduce the behaviour:

Create MORE THAN ONE src csproj with code
Create MORE THAN ONE test csproj with tests that target the src code
Follow instructions in example provided
Either display coverage in a src file or set coverage gutters to watch

Expected behaviour All respective src code lines should indicate accurate code coverage

Screenshots None, let me know if any screenshots are helpful!

Desktop (please complete the following information):

OS: Windows 11
Extension Version 2.10.4
VSCode Version 1.77.3

Additional context The logs seem to indicate that the extension is indeed picking up all of the lcov.info files (removed sensitive file paths):

[1683140467142][coverageservice]: Loading 4 file(s)
[1683140467142][coverageservice]: c:\Users\joshua.nelson\Source\{redacted}.Tests\lcov.info,c:\Users\joshua.nelson\Source\{redacted}.Tests\lcov.info,c:\Users\joshua.nelson\Source\{redacted}.Tests\lcov.info,c:\Users\joshua.nelson\Source\{redacted}.Tests\lcov.info
[1683140467143][coverageservice]: Loaded 4 data file(s)
[1683140467148][coverageservice]: Caching 100 coverage(s)
[1683140467148][coverageservice]: READY
[1683140467148][coverageservice]: RENDERING

ryanluker commented 1 year ago

@Foshkey Thanks for submitting an issue!

You might want to look into using the multi-root workspace functionality (see link from the example project or this link to vscode's docs).

This will allow you to have multiple "folders" or projects open at one time and have the coverage be accessible across different projects.

We can triage and investigate further if it is still an issue for you afterwords.

Foshkey commented 1 year ago

Thanks for the response!

Looks like it's still an issue. To be clear, our github repo is structured like so:

root
|- src
  |- Project1
    |- Project1.csproj
  |- Project2
    |- Project2.csproj
  |- Project3
    |- Project3.csproj
  |- Project4
    |- Project4.csproj
|- tests
  |- unit
    |- Project1.Tests
      |- Project1.Tests.csproj
    |- Project2.Tests
      |- Project2.Tests.csproj
    |- Project3.Tests
      |- Project3.Tests.csproj
    |- Project4.Tests
      |- Project4.Tests.csproj

And opening the root github folder is how most devs operate their workspace.

But, I tried adding all of the src project folders and test project folders to the workspace as separate folders like you suggested, but still the same issue that code gutters indicate no code coverage when all test projects are configured to output code coverage (but still works with just one project)

Foshkey commented 1 year ago

A coworker of mine came up with the suggestion to merge the coverage results with the following test runner arguments:

/p:CollectCoverage=true /p:CoverletOutputFormat=lcov%2cjson /p:CoverletOutput=../CoverageResults/ /p:MergeWith=../CoverageResults/coverage.json -m:1

And then add the setting for coverage gutters to pick up this file:

"coverage-gutters.coverageFileNames": [
    "coverage.info",
    "lcov.info",
    "cov.xml",
    "coverage.xml",
    "jacoco.xml",
    "coverage.cobertura.xml"
  ]

And this worked for me! Might be worth adding this to the docs for when people use multiple test projects. I didn't know that merging coverage results into one file was a thing!

ryanluker commented 1 year ago

A coworker of mine came up with the suggestion to merge the coverage results with the following test runner arguments:
/p:CollectCoverage=true /p:CoverletOutputFormat=lcov%2cjson /p:CoverletOutput=../CoverageResults/ /p:MergeWith=../CoverageResults/coverage.json -m:1
And then add the setting for coverage gutters to pick up this file:
"coverage-gutters.coverageFileNames": [
    "coverage.info",
    "lcov.info",
    "cov.xml",
    "coverage.xml",
    "jacoco.xml",
    "coverage.cobertura.xml"
  ]
And this worked for me! Might be worth adding this to the docs for when people use multiple test projects. I didn't know that merging coverage results into one file was a thing!

Excellent, thanks for finding that fix! Definitely we should add that to the readme / docs as folks with larger projects probably run into this all the time.

pabloandrade-ah commented 1 year ago

A coworker of mine came up with the suggestion to merge the coverage results with the following test runner arguments:
/p:CollectCoverage=true /p:CoverletOutputFormat=lcov%2cjson /p:CoverletOutput=../CoverageResults/ /p:MergeWith=../CoverageResults/coverage.json -m:1
And then add the setting for coverage gutters to pick up this file:
"coverage-gutters.coverageFileNames": [
    "coverage.info",
    "lcov.info",
    "cov.xml",
    "coverage.xml",
    "jacoco.xml",
    "coverage.cobertura.xml"
  ]
And this worked for me! Might be worth adding this to the docs for when people use multiple test projects. I didn't know that merging coverage results into one file was a thing!

what is the -m:1 for?

ryanluker commented 1 year ago

what is the -m:1 for?

Probably related to the merging and keeping things in sequential order (based on the docs I found below).

https://github.com/coverlet-coverage/coverlet/blob/cae57328fcf29e3f8db9c3b2b46f07af09b6e216/Documentation/Examples/MSBuild/MergeWith/HowTo.md?plain=1#L12-L20

valters-tomsons commented 2 months ago

I am facing the same issue with an identical solution/project structure as OP. Projects I'm working on use VSTest integration for coverage, not MSBuild, so report merging is not an option.

Judging from extension logs, the coverage files get detected correctly, but the extension only recognizes the first project's report.

[1723717634376][coverageservice]: Loading 3 file(s)
[1723717634376][coverageservice]: /Users/user/Source/SolutionFolder/Tests/TestProject1/TestResults/98df3ba7-86cd-4cf2-9ea7-0e4096be583c/coverage.cobertura.xml,/Users/user/Source/SolutionFolder/Tests/TestProject2/TestResults/d9c8138a-11d8-4da1-aeb3-337590014f57/coverage.cobertura.xml,/Users/user/Source/SolutionFolder/Tests/TestProject3/TestResults/af06b42e-9bec-4722-81a9-af5fdfd17f8f/coverage.cobertura.xml
[1723717634378][coverageservice]: Loaded 3 data file(s)
[1723717634437][coverageservice]: Caching 115 coverage(s)
[1723717634437][coverageservice]: READY
[1723717634437][coverageservice]: RENDERING

(solution & project path names changed)

Note: As a workaround, you can delete TestResults for all projects and run the coverage collection on a single project at a time. Makes the extension usable, but obviously suboptimal.

ryanluker commented 2 months ago

I am facing the same issue with an identical solution/project structure as OP. Projects I'm working on use VSTest integration for coverage, not MSBuild, so report merging is not an option.

Judging from extension logs, the coverage files get detected correctly, but the extension only recognizes the first project's report.
[1723717634376][coverageservice]: Loading 3 file(s)
[1723717634376][coverageservice]: /Users/user/Source/SolutionFolder/Tests/TestProject1/TestResults/98df3ba7-86cd-4cf2-9ea7-0e4096be583c/coverage.cobertura.xml,/Users/user/Source/SolutionFolder/Tests/TestProject2/TestResults/d9c8138a-11d8-4da1-aeb3-337590014f57/coverage.cobertura.xml,/Users/user/Source/SolutionFolder/Tests/TestProject3/TestResults/af06b42e-9bec-4722-81a9-af5fdfd17f8f/coverage.cobertura.xml
[1723717634378][coverageservice]: Loaded 3 data file(s)
[1723717634437][coverageservice]: Caching 115 coverage(s)
[1723717634437][coverageservice]: READY
[1723717634437][coverageservice]: RENDERING
(solution & project path names changed)

Note: As a workaround, you can delete TestResults for all projects and run the coverage collection on a single project at a time. Makes the extension usable, but obviously suboptimal.

Hmm the extension should find all the coverage reports within the open directory, are you able to open the project at a higher folder levels, so the extension can see the other reports?

The example project in the repo has the by default if you need an example. https://github.com/ryanluker/vscode-coverage-gutters/tree/master/example

ryanluker / vscode-coverage-gutters

Multiple test projects #404