search

chapter-three / next-drupal

Next.js for Drupal has everything you need to build a next-generation front-end for your Drupal site: SSG, SSR, and ISR, Multi-site, Authentication, Webforms, Search API, I18n and Preview mode (works with JSON:API and GraphQL).
https://next-drupal.org
MIT License
610 stars 173 forks source link

code coverage reporting is flakey #740

Open JohnAlbin opened 3 months ago

JohnAlbin commented 3 months ago

Package containing the bug

next-drupal (NPM package)

Describe the bug

If multiple jest files are run, the code coverage of previously-run tests can be reset during the Jest test run and erroneously report a smaller code coverage value than is truly covered. This faulty report is not consistently reported; sometimes the true coverage value is reported.

Expected behavior

The code coverage report is always consistent and accurate instead of frequently reporting missing coverage for code that is obviously covered by existing tests.

Steps to reproduce:

  1. Run yarn test multiple times making slight code changes each time that shouldn't change the coverage value.
  2. Notice the coverage value vacillate.
  3. 😢

Additional context

See https://github.com/jestjs/jest/issues/14766 and https://github.com/nodejs/node/issues/51251

JohnAlbin commented 3 months ago

In https://github.com/nodejs/node/issues/51251, someone mentions that the bug is related to the version of Node being used.

20.9.0 just works 20.10.0 fails

It might also be a bug in the collect-v8-coverage packages used by Jest. https://github.com/SimenB/collect-v8-coverage/pull/235

JohnAlbin commented 3 months ago

GitHub CI is currently using node v18.20.1 where the code coverage is flakey and occasionally failing.

And it looks like v18 versions that were released before v20.10.0 was released (on 2023-11-22) might work. v18.18.x was released before that.

kmccammon commented 3 months ago

I posted this originally on: https://github.com/nodejs/node/issues/51251

Reposting here in case it's helpful...

I have been facing the same issue: my coverage report was 100% with 20.9.0, but as of 20.10.0, it now reports less than 100% coverage and somewhat randomly flags lines not covered.

I noticed that I get the same coverage reporting flakiness with 18.20.2 (I initially saw this issue when my GitHub actions started to fail). Coverage works fine with 18.19.1.

So there seems to be a change between 18.19.1 and 18.20.0 that causes the issue as well as between 20.9.0 to 20.10.0.

This is now causing my GitHub CI/CD actions to fail since GitHub is using 18.20.2.

JohnAlbin commented 3 months ago

I tested v18.19 and it does not appear to be flakey.

JohnAlbin commented 3 months ago

@kmccammon Yep! That's exactly what I noticed as well. I've changed this project's .nvmrc to use v18.19 for now. Thanks for confirming!

I'm leaving this issue open until we can revert the work-around.