vitest-dev / vitest

Next generation testing framework powered by Vite.
https://vitest.dev
MIT License
13.19k stars 1.19k forks source link

--retry=2 gives wrong error: toMatchInlineSnapshot cannot be called multiple times at the same location #6589

Open lydell opened 2 months ago

lydell commented 2 months ago

Describe the bug

When you use the --retry=n flag (where n > 0), inline snapshots that succeeded on the first try fail on the retries with toMatchInlineSnapshot cannot be called multiple times at the same location. This means that if you precede a flaky assertion with an inline snapshot, the --retry flag does not work.

It sounds like the some state that toMatchInlineSnapshot carries needs to be reset between retries.

I later learned that --retry=1 is enough – it’s the number of retries not the number of total tries.

Reproduction

https://github.com/lydell/vitest-inline-snapshot-retry

import {expect, test} from "vitest";

test("demo", () => {
  // First an up-to-date snapshot.
  // Unfortunately, this fails on the second retry.
  expect("a").toMatchInlineSnapshot(`"a"`);

  // Then an assertion of a flaky function.
  expect(flakyFunction()).toBe(2);
});

// Imagine this being a flaky function that only sometimes gives the correct value.
// In this demo, we make it always succeed the second time. Which means that if you
// run with `--retry=2` the test should pass. However, since the snapshot fails on
// the second attempt, the test never passes. Comment out the snapshot test above
// to see it succeed.
let flakyCounter = 1;
function flakyFunction() {
  return flakyCounter++;
}

System Info

System:
    OS: macOS 14.7
    CPU: (10) arm64 Apple M1 Pro
    Memory: 3.02 GB / 32.00 GB
    Shell: 3.7.1 - /opt/homebrew/bin/fish
  Binaries:
    Node: 22.8.0 - ~/.local/share/nvm/v22.8.0/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.8.2 - ~/.local/share/nvm/v22.8.0/bin/npm
    pnpm: 9.9.0 - /opt/homebrew/bin/pnpm
    bun: 1.1.27 - /opt/homebrew/bin/bun
  Browsers:
    Chrome: 129.0.6668.71
    Safari: 18.0
  npmPackages:
    vitest: 2.1.1 => 2.1.1

Used Package Manager

npm

Validations

hi-ogawa commented 2 months ago

toMatshSnapshot is known to have an issue with retry https://github.com/vitest-dev/vitest/issues/5312, but the recent validation made retry not work with toMatshInlineSnapshot as well (though technically there was an issue with retry and toMatshInlineSnapshot in some cases).

Both could be fixed by the same approach mentioned in https://github.com/vitest-dev/vitest/issues/5312#issuecomment-2176222629