bcoe / c8

output coverage reports using Node.js' built in coverage
ISC License
2k stars 91 forks source link

c8 + vitest: Some suites get covered other suites ...dont? #545

Open osher opened 1 month ago

osher commented 1 month ago

c8 + vitest: Some suites get covered other suites ...dont?

I'm working on modernizing and open-sourcing some platform... It's test strategy and test suite is built of two layers:

In the old setup I used nyc, but after modernizing it to recent version of node and refactoring from require to import I could not get nyc to produce coverage for the e2e suites. Then I found about c8.

Astonishingly, c8 produced coverage for the e2e right out of the box with zero configuration. However, evidently, I was too fast to rejoice: It does not produce coverage for the unit tests... 🫨

The opened codebase is still young and does not take long to install or run - so I hope it can be used as a reproduction scenario...

Reproduction

  1. Clone https://gitlab.com/prunjs/prun-core, use branch c8-bug-report.

  2. install (npm i)

  3. "vanilla" run - npm -w ecosystem/prun run cover image

    • Note that 3 suites have run: 1 e2e and 2 unit tests.
  4. run only the e2e suites: npm -w ecosystem/prun run cover -- test/e2e

    • expect: coverage numbers to be lesser than the vanilla run
    • found: coverage are exactly like the "vanilla" run (i.e the "vanilla" run gathers coverage only from the e2e part)
  5. run only unit tests - npm -w ecosystem/prun run cover -- test/unit

    • expect: some coverage to be produced
    • found: The only covered file is vitest.config.js... no coverage is reported for any of the project files image

I believe I could do a mish-mash where I ask both tools to produce coverage report in json for the parts it covers successfully, and ask either of the tools to merge them, but that would be a patch. I thought to ask here before I embark on something weird like that...

I hope it's just something silly I'm missing... donno...

🙏 thanks in advance 🙏

osher commented 1 month ago

@bcoe - I'd really appreciate it if you could take a look, or direct me to who can help

Thanks for all the good work! :)

cenfun commented 1 month ago

You should use vitest coverage directly instead of c8 + vitest

vitest run --coverage

or c8 + mocha (or node test runner)