AVA: snapshot file is corrupted

boneskull commented 8 months ago

Issue description or question

Hi again,

Thanks for your help with #3336. I've got some of my tests running now (though the ones that try to capture process.stdout and process.stderr are failing, which I'm considering "too fancy" to run in Wallaby). However, my snapshot tests are failing:

3 failing tests, 3 passing  Launch Coverage & Test Explorer | Search Tests

  generatePolicy - PolicyGenerator - inspectCompartmentMap() is stable [28 ms] Debug test | Open test story
     
    Could not compare snapshot 
    The snapshot file is corrupted. 
     
    File path: ./packages/endomoat/test/snapshots/policy-generator.spec.js.snap 
     
    Please run AVA again with the --update-snapshots flag to recreate it. 
      at packages/endomoat/test/policy-generator.spec.js:73:3

  generatePolicy - kitchen sink [42 ms] Debug test | Open test story
     
    Could not compare snapshot 
    The snapshot file is corrupted. 
     
    File path: ./packages/endomoat/test/snapshots/policy-generator.spec.js.snap 
     
    Please run AVA again with the --update-snapshots flag to recreate it. 
      at packages/endomoat/test/policy-generator.spec.js:35:3

  generatePolicy - loadCompartmentMap is stable [62 ms] Debug test | Open test story
     
    Could not compare snapshot 
    The snapshot file is corrupted. 
     
    File path: ./packages/endomoat/test/snapshots/policy-generator.spec.js.snap 
     
    Please run AVA again with the --update-snapshots flag to recreate it. 
      at packages/endomoat/test/policy-generator.spec.js:25:3

These tests pass when run from the command line.

I note that the snapshot files are not the same size, which seems weird. Instrumented:

-rw-r--r--  1 boneskull  staff  6782 Feb 28 14:03 instrumented/packages/endomoat/test/snapshots/policy-generator.spec.js.snap

vs original:

-rw-r--r--  1 boneskull  staff  3743 Feb 28 13:34 packages/endomoat/test/snapshots/policy-generator.spec.js.snap

If you can't reproduce, you're welcome to try with https://github.com/LavaMoat/LavaMoat/pull/969 (though I should warn you, it is not so straightforward to get running). The Wallaby config I'm using is not under version control, but you can derive it from the diagnostics.

Wallaby diagnostics report

https://gist.github.com/boneskull/6b8b81761b40b9daf2ad4944fe4f9e48

Note: I'm using symlinkNodeModules: true in my config because I have npm link-ed some dependencies needed to run my tests. Could this be part of the problem?

smcenlly commented 8 months ago

We're still investigating your issue and working on a fix. There are definitely some problems with Wallaby's AVA snapshot support with ESM. We will likely have a fix for you to try by the end of our day tomorrow.

I've got some of my tests running now (though the ones that try to capture process.stdout and process.stderr are failing, which I'm considering "too fancy" to run in Wallaby)

Wallaby uses process.stdout and process.stderr for some interprocess communication, so it's possible that these will cause problems. Let's see after we've provided an updated version of Wallaby that captures snapshots with ESM.

Note: I'm using symlinkNodeModules: true in my config because I have npm link-ed some dependencies needed to run my tests. Could this be part of the problem?

I don't think this is the cause of your problem.

Please note that you will also need to include your snapshot files as files patterns in your Wallaby configuration.

smcenlly commented 8 months ago

There were a few problems affecting snapshots when using the latest version of AVA. There were different issues for both commonjs projects and ESM projects.

The latest version of Wallaby core, v1.0.1532 fixes these issues.

As mentioned above, you will need to ensure that your Wallaby configuration is configured to include your snapshots. For example:

// @ts-check

/** @type {import('wallabyjs').IWallabyConfig} */
module.exports = {
  files: [
    'packages/endomoat/package.json',
    'packages/endomoat/src/**/*.js',
    'packages/endomoat/test/helpers.js',
    'packages/endomoat/test/snapshots/**/*',
    { pattern: 'packages/endomoat/test/fixture/**/*', instrument: false },
  ],
  tests: [
    'packages/endomoat/test/*.spec.js',
    'packages/endomoat/test/*.md', // IMPORTANT - need to include .md snapshot files
    { 
      pattern: 'packages/endomoat/test/*.snap', // IMPORTANT - need to include .snap snapshot files
      binary: true, // IMPORTANT - or will get snapshot file is corrupted
    }, 
    { pattern: 'packages/endomoat/test/scenarios.spec.js', ignore: true },
  ],
  runMode: 'onsave',
  testFramework: 'ava',
  symlinkNodeModules: true,
  worker: {
    restart: true,
  },
  env: {
    type: 'node',
    params: {
      runner: '--experimental-vm-modules',
    },
  },
}

boneskull commented 8 months ago

This fixes the problem. Thank you!

boneskull commented 8 months ago

@smcenlly Is worker a typo or is it an alias for workers?

boneskull commented 8 months ago

I spoke too soon; it still looks broken. I copy/pasted yr config and updated it slightly to reflect some changes:

// @ts-check

/** @type {import('wallabyjs').IWallabyConfig} */
module.exports = {
  files: [
    'packages/endomoat/package.json',
    'packages/endomoat/src/**/*.js',
    'packages/endomoat/test/*-util.js',
    'packages/endomoat/test/scenario-runner.js',
    'packages/endomoat/test/**/snapshots/**/*', 
    { pattern: 'packages/endomoat/test/**/fixture/**/*', instrument: false },
  ],
  tests: [
    'packages/endomoat/test/**/*.spec.js',
    'packages/endomoat/test/**/snapshots/*.md', // IMPORTANT - need to include .md snapshot files
    {
      pattern: 'packages/endomoat/test/**/snapshots/*.snap', // IMPORTANT - need to include .snap snapshot files
      binary: true, // IMPORTANT - or will get snapshot file is corrupted
    },
    { pattern: 'packages/endomoat/test/scenarios.spec.js', ignore: true },
  ],
  runMode: 'onsave',
  testFramework: 'ava',
  symlinkNodeModules: true,
  workers: {
    restart: true,
  },
  env: {
    type: 'node',
    params: {
      runner: '--experimental-vm-modules',
    },
  },
}

(I think worker may have been my typo originally)

Above config gives the same problems under AVA v5.3.1. I note that packages/endomoat/test/**/snapshots/**/* in the files array winds up duplicated in the tests array as per the Wallaby console:

http://localhost:51245
[Error] File/pattern packages/endomoat/test/policy-gen/snapshots/policy-gen.spec.js.md is specified in the files list but also found in the tests list: packages/endomoat/test/**/snapshots/*.md.

[Error]   If it's a test, make sure to exclude it from the files list, for example by adding { pattern: 'packages/endomoat/test/**/snapshots/*.md', ignore: true } to the files list.

However, if I remove that line from the files array, I get many:

[Error] Runtime error: TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".md" for ./packages/endomoat/test/policy-gen/snapshots/policy-generator.spec.js.md
[Error]     at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:160:9)
[Error]     at defaultGetFormat (node:internal/modules/esm/get_format:203:36)
[Error]     at defaultLoad (node:internal/modules/esm/load:141:22)
[Error]     at async nextLoad (node:internal/modules/esm/hooks:865:22)
[Error]     at async nextLoad (node:internal/modules/esm/hooks:865:22)
[Error]     at async Hooks.load (node:internal/modules/esm/hooks:448:20)
[Error]     at async handleMessage (node:internal/modules/esm/worker:196:18)

and

[ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".snap" for ./packages/endomoat/test/policy-gen/snapshots/policy-generator.spec.js.snap
[Error]     at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:160:9)
[Error]     at defaultGetFormat (node:internal/modules/esm/get_format:203:36)
[Error]     at defaultLoad (node:internal/modules/esm/load:141:22)
[Error]     at async nextLoad (node:internal/modules/esm/hooks:865:22)
[Error]     at async nextLoad (node:internal/modules/esm/hooks:865:22)
[Error]     at async Hooks.load (node:internal/modules/esm/hooks:448:20)
[Error]     at async MessagePort.handleMessage (node:internal/modules/esm/worker:196:18)

If I upgrade AVA to latest (v6.1.2), I still get warnings about .md and .snap paths in both files and tests in my console, but the snapshot tests all fail with this instead (after regenerating them):

 PolicyGenerator - buildModuleRecords() is stable [43 ms] Debug test | Open test story
     
    Could not compare snapshot 
      at packages/endomoat/test/policy-gen/policy-generator.spec.js:23:3

Anyway, here's another diagnostic. I'd prefer not to have to upgrade AVA, but I understand it may be a maintenance headache on your end.

smcenlly commented 8 months ago

My apologies, I didn't actually test my configuration with your project and it appears there I had introduced some errors that I didn't get in my sample project I was using.

I've set up a sample project based on your latest diagnostics report that has the same dependencies and it appears to work with both ava@5.3.1 and ava@6.1.2.

Can you please update your Wallaby configuration file as shown below, I think it will work for you:

// @ts-check

/** @type {import('wallabyjs').IWallabyConfig} */
module.exports = {
  files: [
    'packages/endomoat/package.json',
    'packages/endomoat/src/**/*.js',
    'packages/endomoat/test/*-util.js',
    'packages/endomoat/test/scenario-runner.js',
+   'packages/endomoat/test/**/snapshots/*.md', // IMPORTANT - need to include .md snapshot files
+   {
+     pattern: 'packages/endomoat/test/**/snapshots/*.snap', // IMPORTANT - need to include .snap snapshot files
+     binary: true, // IMPORTANT - or will get snapshot file is corrupted
+   },
-   'packages/endomoat/test/**/snapshots/**/*'
    { pattern: 'packages/endomoat/test/**/fixture/**/*', instrument: false },
  ],
  tests: [
    'packages/endomoat/test/**/*.spec.js',
-   'packages/endomoat/test/**/snapshots/*.md', // IMPORTANT - need to include .md snapshot files
-   {
-     pattern: 'packages/endomoat/test/**/snapshots/*.snap', // IMPORTANT - need to include .snap snapshot files
-     binary: true, // IMPORTANT - or will get snapshot file is corrupted
-   },
    { pattern: 'packages/endomoat/test/scenarios.spec.js', ignore: true },
  ],
  runMode: 'onsave',
  testFramework: 'ava',
  symlinkNodeModules: true,
  workers: {
    restart: true,
  },
  env: {
    type: 'node',
    params: {
      runner: '--experimental-vm-modules',
    },
  },
}

smcenlly commented 8 months ago

@boneskull - did this fix your problem?

smcenlly commented 7 months ago

Closing this issue as we didn't hear back.

wallabyjs / public

AVA: snapshot file is corrupted #3337

Issue description or question

Wallaby diagnostics report