Incorrect test results on *first* run after installing node_modules

kwojcik commented 2 months ago

Describe the bug

Using browser mode and either v8 or istanbul coverage, the test run fails the first time after installing node_modules with error Failed to fetch dynamically imported module: http://localhost:5173/@id/@vitest/coverage-v8/browser?import. This is currently blocking me from upgrading because tests fail 100% of the time in CI since it's a fresh node_modules.

> coverage
> vitest run --coverage

 RUN  v2.1.0-beta.6 /Users/kevinwojcik/ontra/eng/vitestsourcemapbug
      Browser runner started at http://localhost:5173/
      Coverage enabled with v8

5:07:40 PM [vite] ✨ new dependencies optimized: @vitest/coverage-v8/browser
5:07:40 PM [vite] ✨ optimized dependencies changed. reloading
5:07:41 PM [vite] ✨ new dependencies optimized: react/jsx-dev-runtime
5:07:41 PM [vite] ✨ optimized dependencies changed. reloading
stderr | test/basic.test.tsx > Link changes the state when hovered
Warning: `ReactDOMTestUtils.act` is deprecated in favor of `React.act`. Import `act` from `react` instead of `react-dom/test-utils`. See https://react.dev/warnings/react-dom-test-utils for more info.
 ✓ test/basic.test.tsx (1)

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
TypeError: Failed to fetch dynamically imported module: http://localhost:5173/@id/@vitest/coverage-v8/browser?import
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 Test Files  1 passed (1)
      Tests  1 passed (1)
     Errors  1 error
   Start at  17:07:39
   Duration  2.11s (transform 0ms, setup 125ms, collect 213ms, tests 13ms, environment 0ms, prepare 45ms)

 % Coverage report from v8
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |
 Link.tsx |       0 |        0 |       0 |       0 | 1-32
----------|---------|----------|---------|---------|-------------------

AFAICT, this was introduced in 2.1.0-beta.1. It works fine in 2.0.5, but look at how basic.test.tsx gets run twice (see below)

> coverage
> vitest run --coverage

 RUN  v2.0.5 /Users/kevinwojcik/ontra/eng/vitestsourcemapbug
      Browser runner started at http://localhost:5173/
      Coverage enabled with istanbul

5:11:22 PM [vite] ✨ new dependencies optimized: react/jsx-dev-runtime
5:11:22 PM [vite] ✨ optimized dependencies changed. reloading
stderr | test/basic.test.tsx > Link changes the state when hovered
Warning: `ReactDOMTestUtils.act` is deprecated in favor of `React.act`. Import `act` from `react` instead of `react-dom/test-utils`. See https://react.dev/warnings/react-dom-test-utils for more info.
 ✓ test/basic.test.tsx (1)
5:11:23 PM [vite] ✨ new dependencies optimized: @vitest/coverage-istanbul
stderr | test/basic.test.tsx > Link changes the state when hovered
Warning: `ReactDOMTestUtils.act` is deprecated in favor of `React.act`. Import `act` from `react` instead of `react-dom/test-utils`. See https://react.dev/warnings/react-dom-test-utils for more info.
 ✓ test/basic.test.tsx (1)
 ✓ test/foo.test.ts (1000)

 Test Files  2 passed (2)
      Tests  1001 passed (1001)
   Start at  17:11:21
   Duration  2.56s (transform 0ms, setup 346ms, collect 45ms, tests 38ms, environment 0ms, prepare 92ms)

 % Coverage report from istanbul
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files |     100 |       50 |     100 |     100 |
 Link.tsx |     100 |       50 |     100 |     100 | 22
----------|---------|----------|---------|---------|-------------------

Reproduction

branch: https://github.com/kwojcik/vitestsourcemapbug/tree/coverageCrashRepro

git clone git@github.com:kwojcik/vitestsourcemapbug.git
cd vitestsourcemapbug
rm -r node_modules; npm install --force && npm run coverage

System Info

System:
    OS: macOS 13.6.7
    CPU: (12) arm64 Apple M2 Max
    Memory: 10.74 GB / 64.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.20.4 - ~/.asdf/installs/nodejs/18.20.4/bin/node
    Yarn: 1.22.21 - /opt/homebrew/bin/yarn
    npm: 10.7.0 - ~/.asdf/plugins/nodejs/shims/npm
    pnpm: 9.5.0 - /opt/homebrew/bin/pnpm
    Watchman: 2024.08.12.00 - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 127.0.6533.122
    Firefox Nightly: 127.0
    Safari: 16.6
  npmPackages:
    @vitest/browser: ^v2.1.0-beta.6 => 2.1.0-beta.6
    @vitest/coverage-istanbul: ^v2.1.0-beta.6 => 2.1.0-beta.6
    @vitest/ui: ^v2.1.0-beta.6 => 2.1.0-beta.6
    vite: latest => 5.3.2
    vitest: ^v2.1.0-beta.6 => 2.1.0-beta.6



### Used Package Manager

npm

### Validations

- [X] Follow our [Code of Conduct](https://github.com/vitest-dev/vitest/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitest-dev/vitest/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitest.dev/guide/).
- [X] Check that there isn't [already an issue](https://github.com/vitest-dev/vitest/issues) that reports the same bug to avoid creating a duplicate.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitest-dev/vitest/discussions) or join our [Discord Chat Server](https://chat.vitest.dev).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

sheremet-va commented 2 months ago

There should be no new dependencies optimized logs at all for Vitest (or any Vite-based testing framework) to work correctly. Having @vitest ones is a bug in Vitest. The react one is interesting, maybe we should add well-known dependencies ourselves.

For now, you can add them to your config manually:

export default defineConfig({
  optimizeDeps: {
    include: ['react/jsx-dev-runtime'],
    exclude: ['@vitest/coverage-istanbul'],
  }
})

AriPerkkio commented 2 months ago

I guess this doesn't show up in our own tests or ecosystem ci due to locally linked packages. It would be great to be able to replicate this issue in our tests at least. 🤔

kwojcik commented 2 months ago

@httpete maybe this is what you were experiencing in https://github.com/vitest-dev/vitest/issues/5615?

sheremet-va commented 2 months ago

The @vitest packages should be fixed by #6445

We also now print the warning that Vitest doesn't expect dependencies to be optimized.

Regarding react - please, use @vitejs/plugin-react if you use react in your project. It adds the jsx-transform to include automatically.

vitest-dev / vitest