search

bcoe / c8

output coverage reports using Node.js' built in coverage
ISC License
2k stars 91 forks source link

Coverage information too high for large esbuild produced bundle #502

Open jakebailey opened 11 months ago

jakebailey commented 11 months ago

I can only apologize in advance for this one...

I work on TypeScript, and I've been wanting to set up coverage information for the compiler. c8 does a great job; significantly faster than nyc (of course). I use c8 in my own projects with success.

However, capturing coverage for the compiler produces results that are too high to believe. checker.ts is a 50k line file, and yet its coverage is:

checker.ts | 99.99 | 95.38 | 98.62 | 99.99 | 50577-50578

There's no way that we actually have tested every line of the checker. And indeed, adding a function to a known-dead path (like one ending in Debug.assertNever) indicates that the line is covered when it definitely is not, and the overall count doesn't change.

image

I don't know if I have a good repro for this besides npx c8 npm test -- --no-lint on our repo. Obviously that's huge but I don't really know where to start.

At best, I can observe that npx c8 npm test -- --no-lint --no-bundle works better, i.e. not using esbuild and instead running the tsc output produces a more believable coverage count:

checker.ts  |    97.5 |    95.87 |   98.62 |    97.5 | ...-48684,48686-48687,48761-48762,48 ...
jakebailey commented 11 months ago

One thing that is interesting is that comparing the bundled output HTML page with the unbundled output HTML page, the paths are different. Specifically, the bundled output starts right with compiler/checker.ts, whereas the unbundled output has src/compiler/checker.ts.

jakebailey commented 11 months ago

Yeah, the unbundled output is way better:

image

ericmorand commented 5 months ago

@jakebailey did you happen to solve the issue? I have the same problem with bundled test suite where, as soon as a portion of a source file is covered, the whole file is marked as covered.

jakebailey commented 5 months ago

No, besides using https://github.com/bcoe/c8/pull/521.

ericmorand commented 5 months ago

Thanks a lot. I tried it and it mostly solve the issue. I'll comment there, with a minimal reproducer repository to, hopefully, help tackle the remainder issues.

gerardo-lima-moonfare commented 2 months ago

Me too I was having 100% coverage when running tests using tsx, which uses esbuild under the hood:

tsx --enable-source-maps --test-reporter=spec --test 'src/**/*.spec.ts'

After some tests, I changed to @swc-node register:

node --import @swc-node/register/esm-register --enable-source-maps --test-reporter=spec --test 'src/**/*.spec.ts'

My preliminary conclusion is: the entry points for the tests were the test files, themselves, and this made any non-tested code to be marked as unreachable by eslint (and removed from the report). This would lead to 100% coverage, by definition.

This was not an exhaustive research, but it makes sense to me and the change produced accurate test coverage reports. I would be interested to understand if other people got different results.