search

istanbuljs / nyc

the Istanbul command line interface
https://istanbul.js.org/
ISC License
5.61k stars 360 forks source link

Merging with nyc storybook test coverage with other reports generated by coverageProvider 'istanbul' result in wrong coverage #1559

Open meanstackmax opened 6 months ago

meanstackmax commented 6 months ago

Link to bug demonstration repository https://github.com/meanstackmax/demo-test-coverage/tree/main

Expected Behavior

Vitest and storybook should use the same source map outputted, thus the issue with different columns for the same stmt not affecting merged report

Observed Behavior

I am trying to generate a merged code report. Therefore, I am using for my unit tests vitest with istanbul. For Storybook I am using the addon which also uses istanbul. To merge the reports I am using nyc.

The issue I am having is that the reports of vitest and storybook are showing me different columns for the same stmt. This leads to not correct merging.

For example in one component vitest said that there is a stmt on line 38 start at col 18 and ends on the same line.

            "2": {
                "start": {
                    "line": 38,
                    "column": 18
                },
                "end": {
                    "line": 38,
                    "column": null
                }
            },

However, storybook coverage report says the statement is on line 38 in column 2:

      "3": {
        "start": {
          "line": 38,
          "column": 2
        },
        "end": {
          "line": 38,
          "column": null
        }
      },

Troubleshooting steps

-Tried to add this to package.json, but it didn't work:

"overrides": {
"istanbul-lib-instrument": "6.0.1",
"istanbul-lib-coverage": "3.2.0"
},

Environment Information

  System:
    OS: macOS 14.4.1
    CPU: (8) arm64 Apple M1
    Memory: 122.00 MB / 16.00 GB
  Binaries:
    Node: 18.16.1 - ~/.nvm/versions/node/v18.16.1/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 9.5.1 - ~/.nvm/versions/node/v18.16.1/bin/npm
  npmPackages:
    @babel/core: ^7.23.7 => 7.24.5 
    @vitest/coverage-istanbul: ^1.2.1 => 1.5.3 
    babel-loader: ^9.1.3 => 9.1.3 
    nyc: ^15.1.0 => 15.1.0 
    typescript: ^5.3.3 => 5.4.5
cenfun commented 6 months ago

There are 3 different versions of istanbul-lib-instrument in use after running npm list istanbul-lib-instrument, it should be the cause of the problem.

coverage-demo-app@1.0.0 /demo-test-coverage
├─┬ @storybook/addon-coverage@1.0.1
│ ├─┬ @jsdevtools/coverage-istanbul-loader@3.0.5
│ │ └── istanbul-lib-instrument@4.0.3
│ ├── istanbul-lib-instrument@6.0.2
│ └─┬ vite-plugin-istanbul@3.0.4
│   └── istanbul-lib-instrument@5.2.1
├─┬ @storybook/addon-essentials@7.6.18
│ └─┬ @storybook/addon-docs@7.6.18
│   └─┬ @jest/transform@29.7.0
│     └─┬ babel-plugin-istanbul@6.1.1
│       └── istanbul-lib-instrument@5.2.1
├─┬ @vitest/coverage-istanbul@1.5.3
│ └── istanbul-lib-instrument@6.0.2 deduped
├─┬ jest@29.7.0
│ └─┬ @jest/core@29.7.0
│   └─┬ @jest/reporters@29.7.0
│     └── istanbul-lib-instrument@6.0.2 deduped
└─┬ nyc@15.1.0
  └── istanbul-lib-instrument@4.0.3
meanstackmax commented 5 months ago

@cenfun thank you, after syncing all istanbul-lib-instrument it seems like an issue still present. It looks like it doesn't count on storybook tests. So I did the following:

snake-py commented 2 months ago

@cenfun I am working on this together with @meanstackmax. I am unsure but it seems it is not working even after harmonizing the dependency resolution. Maybe you have any other idea why we are unable to merge/generate the report properly?

cenfun commented 2 months ago

@snake-py Yes, the Istanbul ecosystem depends not only on istanbul-lib-instrument but also on istanbul-lib-source-maps and istanbul-lib-coverage. It is difficult to determine which one is causing the issue.

In fact, I don't have any other idea about Istanbul, but perhaps you could try V8, which is a native V8 engine coverage report. Following are some examples of merging V8 coverage reports with the mcr tool: