[Bug]: v8 coverage lowers when more tests are executed

[TWiStErRob]

Version

29.7.0

Steps to reproduce

1. Clone https://github.com/TWiStErRob/repros/tree/main/jest/v8-flaky-coverage
2. npm install
3. npm test -- --runInBand wrapped.test.js -> tests pass, coverage 100%
4. npm test -- --runInBand -> tests pass, coverage missing lines

Expected behavior

Everything passes and coverage is better or equivalent to when only a single test is executed.

Actual behavior

File that is covered by a single test file to 100% loses coverage when an additional test is executed. The additional test calls the original function.

Additional context

Anything relevant changed in the repro will make it work correctly. So:

• v8 -> babel ✅
• changing the order of execution of tests (sorted.js < -> >) ✅ We used a testSequencer to consistently reproduce the issue, otherwise it was flaky!
• changing the order via removing --runInBand ✅ We also used this to consistently reproduce the issue, otherwise it was flaky!

Environment

System:
    OS: macOS 14.1.2
    CPU: (12) arm64 Apple M2 Pro or (10) arm64 Apple M1 Pro
  Binaries:
    Node: 20.10.0 - /opt/homebrew/opt/node@20/bin/node
    npm: 10.2.3 - /opt/homebrew/opt/node@20/bin/npm
  npmPackages:
    jest: ^29.7.0 => 29.7.0 

Also on github actions ubuntu-latest (so Ubuntu 22.04 and x64).

[kwmhp]

Hi This seems to be the same problem as in my issue https://github.com/jestjs/jest/issues/14764 Have you tested this with Node 20.9.0?

[TWiStErRob]

Yep, confirmed 20.9.0 -> 20.10.0 causes it.

Raised https://github.com/nodejs/node/issues/51251

[ritchieanesco]

+1 also getting code coverage failure after updating to 20.10.0.

[cenfun]

I found that comment following two lines in node_modules/collect-v8-coverage/index.js will fix it.

    //await this.postSession('Profiler.stopPreciseCoverage');

    //await this.postSession('Profiler.disable');

[ritchieanesco]

Verified that commenting out both lines fixes the issue.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[TWiStErRob]

Still an issue.

[emiwidknowit]

still an issue

[jeroenzwart]

Yep, confirmed 20.9.0 -> 20.10.0 causes it.

Raised nodejs/node#51251

Still a issue and same problem with the upgrade

[cenfun]

Seems it can be fixed with await this.postSession('Debugger.enable'); see https://github.com/SimenB/collect-v8-coverage/pull/235/files

[TatyanaVl]

This is still an issue. Is there any progress?

[kmccammon]

I posted this originally on: https://github.com/nodejs/node/issues/51251

Reposting here in case it's helpful...

I have been facing the same issue: my coverage report was 100% with 20.9.0, but as of 20.10.0, it now reports less than 100% coverage and somewhat randomly flags lines not covered.

I noticed that I get the same coverage reporting flakiness with 18.20.2 (I initially saw this issue when my GitHub actions started to fail). Coverage works fine with 18.19.1.

So there seems to be a change between 18.19.1 and 18.20.0 that causes the issue as well as between 20.9.0 to 20.10.0.

This is now causing my GitHub CI/CD actions to fail since GitHub is using 18.20.2.

[TatyanaVl]

@kmccammon - we are facing exactly the same issue.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[TWiStErRob]

Can't call it closed stale without a triage...

[kmccammon]

Can't call it closed stale without a triage...

Agreed, I'm stuck on node version 18.19.1 until this coverage issue is resolved.

[TWiStErRob]

I think we can safely say this project is abandoned: https://github.com/jestjs/jest/issues?q=is%3Aclosed+label%3A%22Needs+Triage%22+label%3AStale

A ton of them are auto-closed without a single comment.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[Vinnl]

not stale

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[Vinnl]

.

[andreclemente]

Also stuck on node version 18.19 or 20.9 until this coverage issue is resolved.

[aperez-sans]

Can confirm still happening; For those of you whom don't need to --runInBand, running with more than one worker thread fixed the particular problem for me.

npx jest -w 2 ...

[andreclemente]

Running with more than one worker did not fix the problem for me.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[TWiStErRob]

.

[jmls]

also hit this. certainly not stale !

Adding the comments worked for me

[zargham-leanix]

@SimenB any chance to pick this up?

[Zacaria]

not stale