How does code coverage actually work?

anilhelvaci commented 2 months ago

Node.js Version

v23.0.0-pre

NPM Version

9.8.1

Operating System

Darwin Anils-MacBook-Pro.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

Subsystem

Other

Description

Context

I recently decided to contribute to Node.js and the most logical place for me to start seemed to improve code coverage. I went to https://app.codecov.io/gh/nodejs/node to look for a file that has a low coverage. I chose domexception.js as it has 58.17% coverage.

What is the problem?

As I explore more of the Node.js project, I focused on test/wpt/test-domexception.js as my primary source of tests to work on.

However, when I run make coverage-run-js I get 58.82% of coverage BUT if I run the test-domexception.js individually as described in Running Coverage my results are close to 100% like below:

What would solve my problem?

Answers to the question below would solve my problem;

When trying to improve test coverage, how does the development cycle look for a regular nodejs dev?
What should be my source of truth when working on improving test coverage?
Does the fact that test-domexception.js being run in a WPTRunner cause additional complexity when it comes to code coverage?

Any sort of assistance would be greatly appreciated. Thanks 🙏

Minimal Reproduction

No response

Output

No response

Before You Submit

[X] I have looked for issues that already exist before submitting this
[X] My issue follows the guidelines in the README file, and follows the 'How to ask a good question' guide at https://stackoverflow.com/help/how-to-ask

RedYetiDev commented 2 months ago

The codecov coverage is different from what is used internally, I'm not sure how they calculate.

If you'd like to improve the coverage of Node.js code, I suggest using the coverage reported by the makefile, as that's what's used in testing.

anilhelvaci commented 2 months ago

Hey @RedYetiDev, thanks a lot for your answer. However, I'm still stuck despite trying make coverage-run-js and comparing its results with NODE_V8_COVERAGE=coverage/tmp tools/test.py test/wpt/test-domexception.js.

make coverage-run-js results with 58.82%
NODE_V8_COVERAGE=coverage/tmp tools/test.py test/wpt/test-domexception.js results with 100%

I could understand if test coverage went up as I include more tests(as make coverage-run-js runs all JS tests). But struggling to understand why it goes down compared to running only one test file(test/wpt/test-domexception.js). I am assuming this has something to with WPT tests are run via their own runner(WPTRunner) but I am not certain. Could use any insights. Thanks

RedYetiDev commented 2 months ago

The WPT test files aren't the best for calculating code coverage, as they run a suite of predefined tests that aren't controlled by Node.js. That could have something to do with it?

Admittedly, that behavior is quite strange, I might investigate it on my own time.

nodejs / help