Code Coverage Issue with Istanbul Provider

aimad-majdou commented 11 months ago

Describe the bug

I am experiencing an issue with code coverage reporting when using the Istanbul provider. Some files that are exported as default are not marked as covered in the coverage report, even though I have tests for them.

Interestingly, when I add an additional export to these files, they are marked as covered:

Notably, this issue does not occur when I switch to using the V8 provider for code coverage. This behavior seems inconsistent, as other files with only a default export do not require an additional export to get covered.

Reproduction

Check this example: https://stackblitz.com/edit/github-w9fjy7-fjohui

As you can see in image below after the test runs in this example, the file covered.tsx gets covered, but the file notCovered doesn't get covered.

The files and their tests are exactly the same, the only difference is that NotCovered is only exported by a default export, but Covered is exported directly and also by a default export.

Also you can see that the file SubFeature like NotCovered is only exported by a default export, but it gets covered.

System Info

System:
    OS: Windows 10 10.0.19045
    CPU: (8) x64 Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
    Memory: 4.46 GB / 15.86 GB
  Binaries:
    Node: 18.12.1 - C:\Program Files\nodejs\node.EXE     
    npm: 8.19.2 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.19041.1266.0), Chromium (117.0.2045.41)
  npmPackages:
    @vitejs/plugin-react: 4.0.3 => 4.0.3
    @vitejs/plugin-react-swc: 3.3.2 => 3.3.2
    @vitest/coverage-istanbul: 0.34.2 => 0.34.2
    @vitest/ui: 0.34.1 => 0.34.1
    vite: 4.4.5 => 4.4.5
    vitest: 0.34.3 => 0.34.3

Used Package Manager

npm

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

sheremet-va commented 11 months ago

I am unable to provide a specific reproduction case for this issue due to its inconsistency and complexity.

Sorry, but do you expect maintainers to come up with reproductions that are inconsistent and complex? It would be nice if you could put your examples in a small reproduction.

github-actions[bot] commented 11 months ago

Hello @aimad-majdou. Please provide a minimal reproduction using a GitHub repository or StackBlitz. Issues marked with need reproduction will be closed if they have no activity within 3 days.

aimad-majdou commented 11 months ago

@sheremet-va Thank you for your feedback and understanding. I absolutely appreciate the importance of providing a minimal, reproducible example.

However, my initial report was aimed at determining whether this issue with default exports in the context of code coverage is a known one within the vitest team or the community. This was an attempt to gauge whether others have encountered a similar behavior, as it could indicate a broader problem or a configuration issue.

I'll work on creating one to ensure that we can pinpoint the issue more precisely and facilitate its resolution. I'll update the issue with the reproduction once it's ready.

Thank you for your patience and guidance.

stackblitz[bot] commented 11 months ago

Start a new pull request in StackBlitz Codeflow.

github-w9fjy7-fjohui (Open reproduction)

aimad-majdou commented 11 months ago

@sheremet-va please check my post edit, I have provided an example of the issue.

sheremet-va commented 11 months ago

Thank you for your collaboration ❤️

AriPerkkio commented 11 months ago

@aimad-majdou could you try following and see if it fixes the issue for you:

npm i -D @vitejs/plugin-react

vite.config.ts:

+import react from '@vitejs/plugin-react';
-import react from '@vitejs/plugin-react-swc';

Or add an unused export to notCovered.tsx exactly as covered.tsx has:

-const NotCovered = ({
+export const NotCovered = ({

Still need to figure out why this is changing the results though.

AriPerkkio commented 11 months ago

Here is minimal reproduction setup without Vitest: https://github.com/AriPerkkio/swc-istanbul-reproduction/blob/main/debug-swc-sourcemaps.mjs

When @swc/core's jsx.transform.react.runtime option is set to 'automatic', the file is marked as uncovered:

-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |       0 |        0 |       0 |       0 |                   
 repro.tsx |       0 |        0 |       0 |       0 |                   
-----------|---------|----------|---------|---------|-------------------

When it's set to classic the file is marked as covered:

-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |     100 |       60 |     100 |     100 |                   
 repro.tsx |     100 |       60 |     100 |     100 | 5-7               
-----------|---------|----------|---------|---------|-------------------

The source maps seem fine but for some reason the istanbul-lib-source-maps is unable to trace sources using them. Debugging this is really time consuming but if anyone is interested, look into these lines: https://github.com/AriPerkkio/swc-istanbul-reproduction/blob/3d58aa24b496143442b45217b7b8f1c4f9ee5312/debug-swc-sourcemaps.mjs#L99-L103

Equivalent ones on Vitest:

https://github.com/vitest-dev/vitest/blob/db1ff438075d34389de5bb4f5cd08370a08337d5/packages/coverage-istanbul/src/provider.ts#L132-L133

My guess is that root cause is somewhere in the source-map package.

garyb01 commented 7 months ago

I am seeing a similar issue using istanbul.

After adding an extra export:

vitest-dev / vitest