Vitest automatic config is running tests in node_modules

[fvanwijk]

Issue description or question

I'm trying to make Wallaby work with an example project to demonstrate a different issue. The automatic configuration results in a passing test, however it also tries to run tests in node_modules.

You see ‌0 failing tests, 1 passing, 17 todo. The 17 tests are in node_modules/cssstyle/lb/parsers_test.js.

The Wallaby console outputs:

​​[Error] Test without test file id, most likely the test did not run, OR the test/test body is in the file that is not in the tests list in the configuration file {​​ ​​[Error] testFileIdCanNotBeDetermined: 'has only valid properties implemented',​​ ​​[Error] id: 100001,​​ ​​[Error] log: undefined,​​ ​​[Error] filesInvolved: []​​ ​​[Error] }​​

Also note that my test passes in Wallaby, but I expect it to fail (run npm test to see). Maybe the vite.config.ts is overridden by Wallaby somehow?

Wallaby diagnostics report

{
  editorVersion: '1.70.0',
  pluginVersion: '1.0.346',
  editorType: 'VSCode',
  osVersion: 'darwin 21.5.0',
  nodeVersion: 'v16.15.1',
  coreVersion: '1.0.1315',
  checksum: 'MDdmZDk3MzU1NzRmZDc1ZTQyNmExZDhkYmFkZmUzMzUsMTY5MTcxMjAwMDAwMCww',
  config: {
    diagnostics: {
      vitest: {
        file: {
          config: '/// <reference types="vitest" />\n' +
            'import { defineConfig } from "vite";\n' +
            'import vue from "@vitejs/plugin-vue";\n' +
            'import components from "unplugin-vue-components/vite";\n' +
            '\n' +
            '// https://vitejs.dev/config/\n' +
            'export default defineConfig({\n' +
            '  plugins: [\n' +
            '    vue(),\n' +
            '    /* Comment this block to fix the test */\n' +
            '    components({\n' +
            '      resolvers: [(name) => ({ name, from: "./@" })],\n' +
            '    }),\n' +
            '    /* End block */\n' +
            '  ],\n' +
            '  test: {\n' +
            '    environment: "jsdom",\n' +
            '    globals: true, // This is required for Wallaby because somehow it runs tests in node_modules/cssstyle\n' +
            '  },\n' +
            '});\n'
        },
        config: {
          allowOnly: true,
          watch: true,
          globals: true,
          environment: 'jsdom',
          threads: true,
          clearMocks: false,
          restoreMocks: false,
          mockReset: false,
          include: [ '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}' ],
          exclude: [ '**/node_modules/**', '**/dist/**', '**/cypress/**', '**/.{idea,git,cache,output,temp}/**' ],
          testTimeout: 5000,
          hookTimeout: 10000,
          teardownTimeout: 1000,
          isolate: true,
          watchExclude: [ '**/node_modules/**', '**/dist/**' ],
          forceRerunTriggers: [ '**/package.json/**', '**/vitest.config.*/**', '**/vite.config.*/**' ],
          update: false,
          reporters: [ 'default' ],
          silent: false,
          ui: false,
          uiBase: '/__vitest__/',
          open: true,
          css: { include: [ {} ] },
          coverage: {
            enabled: false,
            clean: true,
            cleanOnRerun: false,
            reportsDirectory: '<homeDir>/projects/vite-vitest-stubs/coverage',
            excludeNodeModules: true,
            exclude: [
              'coverage/**',
              'dist/**',
              'packages/*/test{,s}/**',
              '**/*.d.ts',
              'cypress/**',
              'test{,s}/**',
              'test{,-*}.{js,cjs,mjs,ts,tsx,jsx}',
              '**/*{.,-}test.{js,cjs,mjs,ts,tsx,jsx}',
              '**/*{.,-}spec.{js,cjs,mjs,ts,tsx,jsx}',
              '**/__tests__/**',
              '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress}.config.{js,cjs,mjs,ts}',
              '**/.{eslint,mocha,prettier}rc.{js,cjs,yml}'
            ],
            reporter: [ 'text', 'html', 'clover' ],
            allowExternal: false,
            extension: [
              '.js',  '.cjs',
              '.mjs', '.ts',
              '.tsx', '.jsx',
              '.vue', '.svelte'
            ],
            tempDirectory: '<homeDir>/projects/vite-vitest-stubs/coverage/tmp'
          },
          fakeTimers: { loopLimit: 10000, shouldClearNativeTimers: true, toFake: [ 'setTimeout', 'clearTimeout', 'setInterval', 'clearInterval', 'setImmediate', 'clearImmediate', 'Date' ] },
          maxConcurrency: 5,
          dangerouslyIgnoreUnhandledErrors: false,
          defines: {},
          root: '<homeDir>/projects/vite-vitest-stubs',
          deps: { inline: [ {}, {}, {}, {}, {}, {}, '@nuxt/test-utils' ], registerNodeLoader: false },
          snapshotOptions: { snapshotFormat: {}, updateSnapshot: 'new' },
          setupFiles: [],
          cache: { dir: '<homeDir>/projects/vite-vitest-stubs/node_modules/.vitest' },
          sequence: {},
          package: {
            version: '0.21.1',
            urls: { hooks: 'file://<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.346/wallaby552b88/runners/node/hooks.mjs' },
            paths: { root: '<homeDir>/projects/vite-vitest-stubs/node_modules/vitest', dist: '<homeDir>/projects/vite-vitest-stubs/node_modules/vitest/dist' }
          }
        }
      }
    },
    testFramework: { version: 'vitest@0.14.0', configurator: 'vitest@0.14.0', reporter: 'vitest@0.14.0', starter: 'vitest@0.14.0', autoDetected: true },
    preserveComments: false,
    extractComments: true,
    files: [
      { pattern: '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}', ignore: true, trigger: true, load: true, file: true },
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/*.*', ignore: false, trigger: true, load: true, order: 1 }
    ],
    tests: [
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}', ignore: false, trigger: true, load: true, test: true, order: 2 }
    ],
    workers: { initial: 1, regular: 1, recycle: false },
    filesWithNoCoverageCalculated: [],
    runAllTestsInAffectedTestFile: false,
    updateNoMoreThanOneSnapshotPerTestFileRun: false,
    compilers: {},
    logLimits: { inline: { depth: 5, elements: 5000 }, values: { default: { stringLength: 200 }, autoExpand: { elements: 5000, stringLength: 8192, depth: 10 } } },
    preprocessors: {},
    maxConsoleMessagesPerTest: 100,
    autoConsoleLog: true,
    delays: { run: 0, edit: 100, update: 0 },
    teardown: undefined,
    hints: {
      ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore/',
      ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
      commentAutoLog: '?',
      testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' }
    },
    automaticTestFileSelection: true,
    runSelectedTestsOnly: false,
    mapConsoleMessagesStackTrace: false,
    extensions: {},
    env: {
      type: 'node',
      params: { runner: '--experimental-loader=file://<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.346/wallaby552b88/runners/node/hooks.mjs' },
      runner: '<homeDir>/Library/Application Support/fnm/node-versions/v16.15.1/installation/bin/node',
      viewportSize: { width: 800, height: 600 },
      options: { width: 800, height: 600 },
      bundle: true
    },
    reportUnhandledPromises: true,
    slowTestThreshold: 75,
    lowCoverageThreshold: 80,
    loose: undefined,
    symlinkNodeModules: true,
    configCode: 'auto.detect#1187933729'
  },
  packageJSON: {
    dependencies: { vue: '^3.2.37' },
    devDependencies: {
      '@testing-library/vue': '6.6.1',
      '@vitejs/plugin-vue': '^3.0.0',
      jsdom: '20.0.0',
      typescript: '^4.6.4',
      'unplugin-vue-components': '0.22.4',
      vite: '^3.0.0',
      vitest: '0.21.1',
      'vue-tsc': '^0.38.4'
    }
  },
  fs: { numberOfFiles: 19 },
  debug: []
}

[tslocke]

Same here, and not sure that it's related to automatic config. I added

    tests: [
      'test/**/*.test.ts'
    ],

to my wallaby.js and selected that configuration - still running tests in node_modules.

(this just started after I went back to automatic updates and got the latest version, after #3041)

[smcenlly]

Thanks for reporting the problem, this has been fixed in the latest version of Wallaby, v1.0.1317.

[fvanwijk]

I have updated the core. I can verify that the tests in node_modules are now ignored. Thanks for the quick fix. Any pointers what causes the issue?

You see ‌"0 failing tests, 1 passing" instead of "0 failing tests, 1 passing, 17 todo". However the test is still passing, while it is failing with Vitest.

Or could this be related to (unexpected behavior in) unplugin-vue-components in combination with vanilla vite, and is the bug not triggered when running via Wallaby?

[smcenlly]

Any pointers what causes the issue?

The problem in Wallaby was caused by a recently introduced bug.

---

If I understand the rest of your comment properly - you have the same problem when running tests using vitest from the CLI? Perhaps try updating your test matching pattern to ignore node_modules in your vitest configuration?