Coverage stays at 100% for components regardless of the existence of any component test

FlorianPhilipp2007 commented 4 months ago

Describe the bug

Description: When the components option in the Nuxt configuration is set to true (enabling automatic global registration of components), the test coverage report shows 100% coverage for all components, regardless of whether tests actually exist for them.

Actual Behavior: The coverage report incorrectly shows 100% coverage for all components, including those without any tests like TestNestedUntested.vue in component directory

Expected Behavior: The coverage report should accurately reflect the coverage for each component. Only TestNested.vue should show 100% coverage, while TestNestedUnested.vue should show 0% coverage or appropriate coverage based on the absence of tests.

Reproduction

https://stackblitz.com/edit/nuxt-starter-gvfsue?file=nuxt.config.ts,components%2Fnested%2FTestNestedUntested.vue

vitest --coverage is not working in StackBlitz. Please download the project and try it locally.

System Info

Nuxt Version: ^3.12.2
@nuxt/test-utils: 3.12.1
vitest: 1.5.2
@vitest/coverage-v8: 1.5.2
start-server-and-test: 1.15.5
vitest-environment-nuxt: 1.0.0
@vue/test-utils: ^2.4.6
happy-dom: ^14.12.3

Used Package Manager

npm

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

AriPerkkio commented 4 months ago

Something is importing the untested TestNestedUntested. V8 is reporting coverage for it. Maybe Nuxt imports all those files?

{
  "scriptId": "745",
  "url": "file:///x/nuxt-starter-gvfsue/components/nested/TestNestedUntested.vue",
  "functions": [
    {
      "functionName": "",
      "ranges": [{ "startOffset": 0, "endOffset": 981, "count": 1 }],
      "isBlockCoverage": true
    },
    {
      "functionName": "",
      "ranges": [{ "startOffset": 13, "endOffset": 981, "count": 1 }],
      "isBlockCoverage": true
    },
    {
      "functionName": "",
      "ranges": [{ "startOffset": 444, "endOffset": 564, "count": 0 }],
      "isBlockCoverage": false
    }
  ]
}

Coverage is also off because source maps from Vue (or Nuxt?) don't look alright: https://evanw.github.io/source-map-visualization/#NzIwAGNvbn.

Looks similar to https://github.com/vitest-dev/vitest/issues/3607. I would recommend to open an issue on Nuxt or Vue.

FlorianPhilipp2007 commented 4 months ago

Something is importing the untested TestNestedUntested. V8 is reporting coverage for it. Maybe Nuxt imports all those files?
{
  "scriptId": "745",
  "url": "file:///x/nuxt-starter-gvfsue/components/nested/TestNestedUntested.vue",
  "functions": [
    {
      "functionName": "",
      "ranges": [{ "startOffset": 0, "endOffset": 981, "count": 1 }],
      "isBlockCoverage": true
    },
    {
      "functionName": "",
      "ranges": [{ "startOffset": 13, "endOffset": 981, "count": 1 }],
      "isBlockCoverage": true
    },
    {
      "functionName": "",
      "ranges": [{ "startOffset": 444, "endOffset": 564, "count": 0 }],
      "isBlockCoverage": false
    }
  ]
}
Coverage is also off because source maps from Vue (or Nuxt?) don't look alright: https://evanw.github.io/source-map-visualization/#NzIwAGNvbn.

Looks similar to #3607. I would recommend to open an issue on Nuxt or Vue.

I raised the problem at nuxt but they think its a vitest bug https://github.com/nuxt/test-utils/issues/891#issuecomment-2230925946

AriPerkkio commented 4 months ago

@FlorianPhilipp2007 could you test if coverage.provider: 'istanbul' works better here? Under the hood it works differently than the default v8 provider. It could help in hiding the untested files, but the report will still be off due to source maps issues.

DenisLug commented 2 months ago

@AriPerkkio, I'm taking over for my colleague. I tested it out with the latest Vitest (2.0.5) and with the Istanbul provider (https://stackblitz.com/edit/nuxt-starter-q9e4id). The coverage report with istanbul looks like this:

As you can see, it collects only the coverage for the utils directory.

When I switch back to v8, the image completely changes with Vitest V2, and it now shows 0% coverage for files that have a test:

AriPerkkio commented 2 months ago

How does Nuxt import those files? Sounds like it's not using the Vite server at all. 🤔

Are there any test runners that can show code coverage of Nuxt projects?

DenisLug commented 2 months ago

I'm not aware of any nuxt-specific test runners. @AriPerkkio, you would say that this is likely a nuxt issue? I would then continue the discussion in https://github.com/nuxt/test-utils/issues/891.

vitest-dev / vitest