Snapshot not comparing

[MartinFerret]

I'm testing the snapshot part of the application. It seems that when you run npx jest, it generates the screenshot of the page easily. But when modifying the page so it does not match, the expected behaviour would be for the test to fail, but it passes.

Test repository here (the failing test with the snapshot is under src/app/components/test) :

https://github.com/MartinFerret/safetest-error-case

Also :

• Why running npm safetest doesnt run the tests ? It only gives files matching.
• The -u or --updateSnapshot flags are not working after npx jest, is it the expected behaviour ?
• How to integrate this windows configuration in order to make our tests to run in ci ?

[kolodny]

So SafeTest is pretty lenient regarding matching snapshots when running locally, in CI it requires exact matching.

If you want you can modify the setup code to the following

setup({
  bootstrappedAt: require.resolve("./src/main.ts"),
  url: "http://localhost:4200/angular-app",
  headless: false,
  matchImageSnapshotOptions: {
    failureThreshold: 0,
    failureThresholdType: 'pixel',
  },
  ciOptions: {
    usingArtifactsDir: "artifacts",
  },
});

Also you'll need to remove this line

Regarding the other items

• I'm not sure what's happening here, this seems to be a problem only in Windows. You should be able to get rid of these flags in the package.json safetest script since the jest.config already has them and it then runs correctly:
  --testMatch '**/*.safetest.{j,t}s{,x}' --setupFilesAfterEnv ./setup-safetest.js --testTimeout=30000
• You need to pass true to matchImageSnapshotOptions.updatePassedSnapshot in the setup code above
• I'm not sure about the Windows part

[MartinFerret]

Thanks for your precisions. I'm running some tests and with the configuration you gave me above, the test fails in every case even when nothing has changed in the UI (available in my last commit).

If I run : npx jest -> Creates me the snapshot. Then : npx jest -> Test failed.

Also, the snapshot doesn't seem to update when taking a new one, maybe due to the error mentionned above.

Thanks a lot for your time :)

[kolodny]

Hmm, I'm not sure, can you try running a vanilla version of this by creating a new project and adding jest, playwright and https://github.com/americanexpress/jest-image-snapshot?tab=readme-ov-file#-usage and then making a breaking screenshot change? It's possible that you're missing something like imagemagick or some other dep that isn't getting bundled with windows. I was able to get a screenshot diff working on my Windows machine on your repo

[MartinFerret]

Yeah I'll try that, it's strange.

So in my repo, with your windows machine, the tests are OK when nothing changes but fail when you change something in the UI ? So from your side, everything looks OK?

Is imagemagick required to run ?

[kolodny]

Yeah, everything looks ok. Maybe try removing the object passed to toMatchImageSnapshot in the tests

[MartinFerret]

I've got rid of the empty passed object, and installed imagemagick and put it in my env path, but nothing changes. I will dig into it, please let me know if think about something.

[MartinFerret]

By implementing it in a CI/CD gitlab, won't this below cause the job to fail ?

The URL is hardcoded, as the pipeline job will be executed in a isolated environment, it will fail obviously. As well we'll need a headless conf to run playwright I guess

I've tested it multiple times with mltiple config but can't proceed any further for now, here is my gitlab job :

[kolodny]

Chromium not being able to be launched is outside the scope of this library and probably something you'll need to bring to the Playwright team, it may be you need to install system requirements but I'm not that knowledgable in that. You'll also need to figure out how to get the config to be different in CI vs locally, SafeTest does this by using the linux bash default expander which won't work on Windows.

One last thing for the image diff error, I see the error is coming from this line, you can try passing runInProcess to the config and see if that gives you more info on the error or just set a breakpoint in the node_modules package and trace it yourself