JSON files nor any report files are created when running tests with compareSnapshot

[JasonFairchild]

After running a test where screenshots are taken nothing for reporting is created, not even when comparisons fail. So when I run npx cypress-image-diff-html-report generate I get [cypress-image-diff-html-report]: Cannot find any report json in this directory ...\component-testing-quickstart-apps\react\my-awesome-app\cypress-image-diff-html-report

Which by all accounts, should be the correct directory I'm expecting it to look in (the default one). I went through all the setup steps, I'm quite certain, so what am I missing here... I have been able to get the old v1 report to get generated, if that is worth anything.

[tamasmagyar]

If I understand correctly this is an issue with cypress-image-diff-js? Report generation is handled by that tool. Source

I noticed that the default json file report file name is empty in the repo. Can you try adding a config file, please? Source. Make sure to have value for FILENAME. I will fix it, if it solves the issue.

If the step above doesn't solve the issue, can you re-create the issue in https://github.com/haim-io/cypress-image-diff please?

[JasonFairchild]

Well, this turned out to be because I was only running the headed or GUI form of cypress (cypress open), and apparently the json report is only generated when running headless (cypress run). Which is rather unfortunate and unintuitive to me and certainly worth pointing out somewhere in the docs, probably on both sides of these NPM packages. I love using cypress with their UI. Basically this issue that had been previously raised.

On the other hand, I've had a lot of problems with little visual differences being caught between headed and headless mode while using cypress-image-diff. Like with elements being in focus or not, small font rendering differences and more. I was able to overcome all of them with browser options and other tricks, but it can be time-consuming. It seems like it would be super handy if there was some high level config option that allowed you to completely turn off the visual test assertions when running in one mode or the other (or at least turned off in headed). That way it could be very feasible to do VRT with this tool, without having to get the test passing in both modes with a single baseline. Perhaps there is a way you could configure it just with Cypress options and how/when it takes screenshots, IDK...

Also I think it is worth calling out that during my troubleshooting of all this, I did try making a cypress-image-diff.config.cjs which worked to designate a custom root directory and report directory, but after I had finally gotten the json report to be created, the cypress-image-diff-report generate command appeared to not respect this and showed this error message: [cypress-image-diff-html-report]: Given reportJsonDir does not exist ...\my-awesome-app\cypress-image-diff-html-report. Make sure you specify a valid reportJsonDir or reportJsonFilePath

So seems like a separate issue, but related and important. I should have captured more of that but I deleted it and went back to the defaults and it is working as long as I run headlessly now. So I don't really feel like recreating that, but I'll at least give you this heads up about the likely issue. This is basically the config I was using:

const config = {
    ROOT_DIR: 'visual-test',
    REPORT_DIR: 'json-report',
    JSON_REPORT: { 
      FILENAME: 'cypress_visual_report',
      // OVERWRITE: false,
    }, 
  };

  module.exports = config;

[tamasmagyar]

The recommended way to use cypress-image-diff-js is running in docker, to avoid environment specific differences. Ref:

Note: Be aware that despite forcing a screenshot resolution to a particular height and width for a test, if this test is run on different infrastructure (i.e a 13" Mac vs PC attached to a 30" monitor), the results will be different. So it's extremely important that you standardize where the tests will run, both locally and CI. One way to handle this, is by running it with docker container or against BrowserStack or alike. This project tests use a docker container to run the tests so it could be used as an example.

Re:

It seems like it would be super handy if there was some high level config option that allowed you to completely turn off the visual test assertions

You can try adjusting FAILURE_THRESHOLD ref

[JasonFairchild]

Yeah, I've experienced many of the difference from running on different OS or other differences in CI vs local as well. I've gotten pushback from devs for having to run things in docker that they feel should be simple things to run directly locally. But if the docker process remained very fast and simple, I think it is a good option. And I suppose adjusting the failure threshold temporarily or through a gitignored file could work well enough too.