vitest unable to get test coverage in vm

[DiamondYuan]

Issue description or question

REPO: https://github.com/ant-design/ant-design-mini/tree/vm

Here is the process I tested

1. Build the project using esbuild
2. use instrumenter
3. in afterEach, merge the test coverage into __VITEST_COVERAGE__ of vitest

I can get the test coverage correctly with vitest run --coverage, but not with wallaby.js.

Expected

Open src/Badge/__tests__/index.test.ts , run wallaby , and get the coverage in src/Badge/index.ts.

Wallaby diagnostics report

{
  editorVersion: '1.81.1',
  pluginVersion: '1.0.365',
  editorType: 'VSCode',
  osVersion: 'darwin 22.5.0',
  nodeVersion: 'v18.17.0',
  coreVersion: '1.0.1464',
  checksum: 'MTdhZDc3MjYwYjQ0MzNmNTQ4MGJiMWUxZWYzMGU3OWYsMTcyNDYzMDQwMDAwMCww',
  config: {
    diagnostics: {
      vitest: {
        file: {
          config: '// vite.config.ts\n' +
            "import { defineConfig } from 'vitest/config';\n" +
            '\n' +
            'export default defineConfig({\n' +
            '  test: {\n' +
            '    watch: true,\n' +
            '    globals: true,\n' +
            "    setupFiles: ['./tests/setup.ts'],\n" +
            '    coverage: {\n' +
            "      provider: 'istanbul',\n" +
            '    },\n' +
            '  },\n' +
            '});\n'
        },
        config: {
          allowOnly: true,
          watch: true,
          globals: true,
          environment: 'node',
          threads: true,
          clearMocks: false,
          restoreMocks: false,
          mockReset: false,
          include: [ '**/*.{test,spec}.?(c|m)[jt]s?(x)' ],
          exclude: [
            '**/node_modules/**',
            '**/dist/**',
            '**/cypress/**',
            '**/.{idea,git,cache,output,temp}/**',
            '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build}.config.*'
          ],
          testTimeout: 5000,
          hookTimeout: 10000,
          teardownTimeout: 10000,
          isolate: true,
          watchExclude: [ '**/node_modules/**', '**/dist/**' ],
          forceRerunTriggers: [ '**/package.json/**', '**/{vitest,vite}.config.*/**', '<homeDir>/projj/github.com/ant-design/ant-design-mini/tests/setup.ts' ],
          update: false,
          reporters: [ 'default' ],
          silent: false,
          hideSkippedTests: false,
          ui: false,
          uiBase: '/__vitest__/',
          open: true,
          css: { include: [], modules: { classNameStrategy: 'stable' } },
          coverage: {
            provider: 'istanbul',
            enabled: false,
            clean: true,
            cleanOnRerun: true,
            reportsDirectory: './coverage',
            exclude: [
              'coverage/**',
              'dist/**',
              'packages/*/test?(s)/**',
              '**/*.d.ts',
              '**/virtual:*',
              '**/__x00__*',
              '**/\x00*',
              'cypress/**',
              'test?(s)/**',
              'test?(-*).?(c|m)[jt]s?(x)',
              '**/*{.,-}{test,spec}.?(c|m)[jt]s?(x)',
              '**/__tests__/**',
              '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build}.config.*',
              '**/.{eslint,mocha,prettier}rc.{?(c|m)js,yml}',
              'tests/setup.ts',
              'tests/setup.ts'
            ],
            reportOnFailure: false,
            reporter: [ [ 'text', {} ], [ 'html', {} ], [ 'clover', {} ], [ 'json', {} ] ],
            extension: [
              '.js',  '.cjs',
              '.mjs', '.ts',
              '.mts', '.cts',
              '.tsx', '.jsx',
              '.vue', '.svelte'
            ]
          },
          fakeTimers: { loopLimit: 10000, shouldClearNativeTimers: true, toFake: [ 'setTimeout', 'clearTimeout', 'setInterval', 'clearInterval', 'setImmediate', 'clearImmediate', 'Date' ] },
          maxConcurrency: 5,
          dangerouslyIgnoreUnhandledErrors: false,
          typecheck: {
            checker: 'tsc',
            include: [ '**/*.{test,spec}-d.?(c|m)[jt]s?(x)' ],
            exclude: [
              '**/node_modules/**',
              '**/dist/**',
              '**/cypress/**',
              '**/.{idea,git,cache,output,temp}/**',
              '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build}.config.*'
            ]
          },
          slowTestThreshold: 300,
          setupFiles: [ '<homeDir>/projj/github.com/ant-design/ant-design-mini/tests/setup.ts' ],
          config: '<homeDir>/projj/github.com/ant-design/ant-design-mini/vite.config.ts',
          defines: {},
          root: '<homeDir>/projj/github.com/ant-design/ant-design-mini',
          mode: [],
          inspect: false,
          inspectBrk: false,
          singleThread: false,
          deps: { moduleDirectories: [ '/node_modules/' ], web: { transformAssets: true, transformCss: true, transformGlobPattern: [] } },
          server: { deps: { inline: [ {}, {}, {}, '@nuxt/test-utils' ], moduleDirectories: [ '/node_modules/' ], cacheDir: 'node_modules/.vitest' } },
          snapshotOptions: { snapshotFormat: {}, updateSnapshot: 'new', snapshotEnvironment: null },
          experimentalVmWorkerMemoryLimit: 1431655765,
          cache: { dir: '<homeDir>/projj/github.com/ant-design/ant-design-mini/node_modules/.vitest' },
          sequence: { hooks: 'parallel' },
          environmentMatchGlobs: [],
          browser: { enabled: false, headless: false, slowHijackESM: true, api: { port: 63315 } },
          testTransformMode: {},
          package: {
            version: '0.34.2',
            urls: { hooks: 'file://<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/runners/node/hooks.mjs' },
            paths: {
              root: '<homeDir>/projj/github.com/ant-design/ant-design-mini/node_modules/vitest',
              dist: '<homeDir>/projj/github.com/ant-design/ant-design-mini/node_modules/vitest/dist'
            }
          }
        }
      }
    },
    testFramework: { version: 'vitest@0.14.0', configurator: 'vitest@0.14.0', reporter: 'vitest@0.14.0', starter: 'vitest@0.14.0', autoDetected: true },
    preserveComments: false,
    extractComments: true,
    files: [
      { pattern: '**/*.{test,spec}.?(c|m)[jt]s?(x)', ignore: true, trigger: true, load: true, file: true },
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build}.config.*', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/*.*', ignore: false, trigger: true, load: true, order: 1 }
    ],
    tests: [
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build}.config.*', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/*.{test,spec}.?(c|m)[jt]s?(x)', ignore: false, trigger: true, load: true, test: true, order: 2 }
    ],
    workers: { initial: 1, regular: 1, recycle: false },
    captureConsoleLog: true,
    filesWithNoCoverageCalculated: [],
    runAllTestsInAffectedTestFile: false,
    updateNoMoreThanOneSnapshotPerTestFileRun: false,
    compilers: {},
    smartStart: [ { test: true, source: false, pattern: '**/*', startMode: 'open' }, { test: false, source: true, pattern: '**/*', startMode: 'edit' } ],
    logLimits: { inline: { depth: 5, elements: 5000 }, values: { default: { stringLength: 200 }, autoExpand: { elements: 5000, stringLength: 8192, depth: 10 } } },
    preprocessors: {},
    maxConsoleMessagesPerTest: 100,
    autoConsoleLog: true,
    delays: { run: 0, edit: 100, update: 0 },
    teardown: undefined,
    hints: {
      ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore|c8 ignore/',
      ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
      commentAutoLog: '?',
      testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' }
    },
    automaticTestFileSelection: true,
    runSelectedTestsOnly: true,
    mapConsoleMessagesStackTrace: false,
    extensions: {},
    env: {
      type: 'node',
      params: { runner: '--experimental-loader=file://<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/runners/node/hooks.mjs' },
      runner: '<homeDir>/.nvm/versions/node/v18.17.0/bin/node',
      viewportSize: { width: 800, height: 600 },
      options: { width: 800, height: 600 },
      bundle: true
    },
    reportUnhandledPromises: true,
    slowTestThreshold: 75,
    lowCoverageThreshold: 80,
    runAllTestsWhenNoAffectedTests: true,
    symlinkNodeModules: undefined,
    configCode: 'auto.detect#2145583041',
    testLocations: [ 'src/Badge/__tests__/index.test.ts' ]
  },
  packageJSON: {
    dependencies: { '@mini-types/alipay': '^3.0.5', 'async-validator': '^4.0.7', dayjs: '^1.11.3', 'fast-deep-equal': '3.1.3', tslib: '2.5.0' },
    devDependencies: {
      '@types/less': '^3.0.3',
      '@typescript-eslint/eslint-plugin': '^5.6.0',
      '@typescript-eslint/parser': '^5.6.0',
      '@vitest/coverage-istanbul': '^0.34.2',
      antd: '^4.20.7',
      'auto-changelog': '^2.4.0',
      chalk: '^4.1.0',
      'clone-deep': '^4.0.1',
      'conventional-changelog-cli': '^2.1.1',
      'copy-to-clipboard': '^3.3.3',
      dumi: '^1.1.35',
      esbuild: '^0.19.2',
      eslint: '^8.4.1',
      'eslint-config-ali': '^13.0.0',
      'eslint-config-prettier': '^8.3.0',
      'eslint-plugin-import': '^2.25.3',
      'extract-changelog-release': '^1.0.2',
      'gh-pages': '^3.0.0',
      glob: '^9.0.1',
      gulp: '^4.0.2',
      'gulp-babel': '^8.0.0',
      'gulp-clean-css': '^4.0.0',
      'gulp-inject-envs': '^1.0.0',
      'gulp-less': '^5.0.0',
      'gulp-rename': '^1.2.3',
      'gulp-typescript': '^6.0.0-alpha.1',
      inquirer: '^8.2.1',
      'istanbul-lib-coverage': '^3.2.0',
      less: '^4.1.2',
      'lint-staged': '^10.0.7',
      merge2: '^1.4.1',
      minidev: '^1.0.7',
      'postcss-less': '^6.0.0',
      'pre-commit': '^1.2.2',
      prettier: '^2.2.1',
      'qrcode.react': '^3.1.0',
      'resize-observer-polyfill': '^1.5.1',
      shallowequal: '^1.1.0',
      stylelint: '^14.1.0',
      'stylelint-config-standard': '^24.0.0',
      typescript: '^4.0.0',
      vitest: '^0.34.2',
      yorkie: '^2.0.0'
    }
  },
  fs: { numberOfFiles: 968 },
  debug: [
    '2023-08-28T02:37:56.300Z config Attempting automatic configuration for angular\n',
    '2023-08-28T02:37:56.302Z angular/cli config Angular CLI not found.\n',
    '2023-08-28T02:37:56.302Z config Finished attempting automatic configuration for angular (2ms)\n',
    '2023-08-28T02:37:56.302Z config Attempting automatic configuration for jest\n',
    "2023-08-28T02:37:56.302Z jest/config Error: Module jest-cli is not found in '<homeDir>/projj/github.com/ant-design/ant-design-mini'.\n" +
      '    at Object.loadJest (<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/server.js:496:1070)\n' +
      '    at n (<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/server.js:184:23339)\n' +
      '    at Object.configure (<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/server.js:184:26449)\n' +
      '    at automaticConfigurationProviders.reduce.Promise.resolve.success (<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/server.js:132:12015)\n' +
      '    at async automaticConfigurationProviders.reduce.Promise.resolve.success (<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/server.js:132:11892)\n' +
      '    at async Config.load (<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/wallaby30e036/server.js:132:11832)\n',
    '2023-08-28T02:37:56.302Z config Finished attempting automatic configuration for jest (0ms)\n',
    '2023-08-28T02:37:56.302Z config Attempting automatic configuration for vitest\n',
    '2023-08-28T02:37:56.304Z vitest/config Detected Vitest (0.34.2).\n',
    '2023-08-28T02:37:56.744Z config Finished attempting automatic configuration for vitest (442ms)\n',
    '2023-08-28T02:37:56.745Z project Wallaby Node version: v18.17.0\n',
    '2023-08-28T02:37:56.745Z project Wallaby config: <homeDir>/projj/github.com/ant-design/ant-design-mini/auto.detect\n',
    '2023-08-28T02:37:56.767Z fs File system starting\n',
    '2023-08-28T02:37:56.858Z fs File system scan completed\n',
    '2023-08-28T02:37:56.863Z project File cache: <homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.365/projects/7da8d8e50dd7c4f1\n',
    '2023-08-28T02:37:56.907Z uiService Listening port 51235\n',
    '2023-08-28T02:37:56.928Z workers Parallelism for initial run: 1, for regular run: 1\n',
    '2023-08-28T02:37:56.928Z workers Starting run worker instance #0\n',
    '2023-08-28T02:37:56.928Z workers Web server is listening at 59621\n',
    '2023-08-28T02:37:56.930Z project Stopping process pool\n',
    '2023-08-28T02:37:56.930Z project File cache is up-to-date, starting full test run\n',
    '2023-08-28T02:37:56.930Z project Test run started; run priority: 3\n',
    '2023-08-28T02:37:56.932Z project Running all tests\n',
    '2023-08-28T02:37:56.938Z project Test run finished\n',
    '2023-08-28T02:37:56.938Z project Test run data re-queued\n',
    '2023-08-28T02:37:57.033Z project Requested to run all tests\n',
    '2023-08-28T02:37:57.035Z project Test run started; run priority: 3\n',
    '2023-08-28T02:37:57.035Z project Running all tests\n',
    '2023-08-28T02:37:57.041Z workers Starting test run, priority: 3\n',
    '2023-08-28T02:37:57.041Z nodeRunner Starting sandbox [worker #0, session #lzggy]\n',
    '2023-08-28T02:37:57.041Z nodeRunner Preparing sandbox [worker #0, session #lzggy]\n',
    '2023-08-28T02:37:57.048Z workers Started run worker instance (delayed) #0\n',
    '2023-08-28T02:37:57.048Z nodeRunner Prepared sandbox [worker #0, session #lzggy]\n',
    '2023-08-28T02:37:57.048Z workers [worker #0, session #lzggy] Running tests in sandbox\n',
    '2023-08-28T02:37:57.425Z workers Scheduling Vitest Run (lzggy): 2023-08-28T02:37:57.412Z\n',
    '2023-08-28T02:37:57.718Z workers [lzggy.1] Loaded unknown number of test(s)\n',
    '2023-08-28T02:37:57.882Z workers [lzggy.1] Test executed: badge overCount true\n',
    '2023-08-28T02:37:57.882Z workers [lzggy.1] Test executed: badge overCount false\n',
    '2023-08-28T02:37:57.889Z fs No metadata for added file found: node_modules/.vitest/results.json\n',
    '2023-08-28T02:37:57.891Z workers Vitest Run Complete (lzggy): 2023-08-28T02:37:57.889Z\n',
    '2023-08-28T02:37:57.892Z workers [lzggy] Run 2 test(s), skipped 0 test(s)\n',
    '2023-08-28T02:37:57.893Z workers [lzggy] Sandbox is responsive, closing it\n',
    '2023-08-28T02:37:57.893Z project Test run finished\n',
    '2023-08-28T02:37:57.893Z project Processed console.log entries\n',
    '2023-08-28T02:37:57.893Z project Processed loading sequences\n',
    '2023-08-28T02:37:57.893Z project Processed executed tests\n',
    '2023-08-28T02:37:57.895Z project Processed code coverage\n',
    '2023-08-28T02:37:57.903Z project Test run result processed and sent to IDE\n',
    '2023-08-28T02:37:57.990Z fs File changed: node_modules/.vitest/results.json\n',
    '2023-08-28T02:37:57.990Z fs No metadata for changed file found: node_modules/.vitest/results.json\n'
  ]
}

[smcenlly]

Thanks for the sample repo.

Wallaby's behavior for your application is actually expected. In your tests, you are using getInstance() from tests/utils.ts where you bypass Vitest's pipeline for processing files with a custom callout to esbuild.buildSync and then subsequently when you run the generated code with vm.runInContext.

This may work for Vitest's code coverage mechanism with istanbul but you will see that it fails the same way as in the Wallaby when you use the default v8 code coverage provider (see coverage report using v8 below):

 % Coverage report from v8
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |   86.11 |     90.9 |     100 |   86.11 |                   
 Stepper      |   82.14 |       90 |     100 |   82.14 |                   
  utils.ts    |   82.14 |       90 |     100 |   82.14 | 47-56             
 _util        |     100 |      100 |     100 |     100 |                   
  fmtEvent.ts |     100 |      100 |     100 |     100 |                   
--------------|---------|----------|---------|---------|-------------------

---

Unfortunately we don't have a way of supporting your current method of loading your code files.

I'm wondering what's the reason for doing this? If you are able to structure your code to load the files without a customer transpiler / and vm context, then Wallaby will start reporting coverage.

[DiamondYuan]

Thanks for the sample repo.

Wallaby's behavior for your application is actually expected. In your tests, you are using getInstance() from tests/utils.ts where you bypass Vitest's pipeline for processing files with a custom callout to esbuild.buildSync and then subsequently when you run the generated code with vm.runInContext.

This may work for Vitest's code coverage mechanism with istanbul but you will see that it fails the same way as in the Wallaby when you use the default v8 code coverage provider (see coverage report using v8 below):

 % Coverage report from v8
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |   86.11 |     90.9 |     100 |   86.11 |                   
 Stepper      |   82.14 |       90 |     100 |   82.14 |                   
  utils.ts    |   82.14 |       90 |     100 |   82.14 | 47-56             
 _util        |     100 |      100 |     100 |     100 |                   
  fmtEvent.ts |     100 |      100 |     100 |     100 |                   
--------------|---------|----------|---------|---------|-------------------

Unfortunately we don't have a way of supporting your current method of loading your code files.

I'm wondering what's the reason for doing this? If you are able to structure your code to load the files without a customer transpiler / and vm context, then Wallaby will start reporting coverage.

Components depend on global my and Componet, that's why I'm executing the code in vm.

Is there any way to expose coverage data to wallaby.js manually?

[smcenlly]

Is there any way to expose coverage data to wallaby.js manually?

Wallaby's code coverage is proprietary and there is currently no way to push code coverage results.

---

Having said that, what you're doing can be done natively from within the testing framework, and I think it would be better to lean into what is natively supported by the testing framework vs. creating your own bespoke mechanism to transform and collect code coverage.

You will need to modify your tests a little, but you can do something like this:

1. Change tests/utils.ts to stub the globals you need

tests/utils.ts

...
// Add below other imports
import { vi } from 'vitest';

...

// Change getInstance() method to:
async function getInstance(
  name: string,
  props: Record<string, any>,
  api?: Record<string, any>
) {
  let result;

  vi.stubGlobal('my', api);
  vi.stubGlobal('Component', (obj) => {
    result = createInstance(obj, props, api, {});
  });

  await import(path.join(__dirname, `../src/${name}/index.ts`))

  return result;
}

This code now uses the vitest stubGlobal method for your my and Component globals.

2. Update your tests to use getInstance() as async instead of synchronous call

src/Badge/tests/index.test.ts

import { getInstance } from '../../../tests/utils';
import { describe, it, expect } from 'vitest';

describe('badge overCount', () => {
  it('badge overCount true', async () => {
    const instance = await getInstance('Badge', {
      text: 102,
    });
    expect(instance.getData().overCount).toBe(true);
  });

  it('badge overCount false', async () => {
    const instance = await getInstance('Badge', {
      text: 75,
    });
    expect(instance.getData().overCount).toBe(false);
  });
});