Feature Suggestion: Applitools Visual Testing

[myspivey]

Hey Folks, my name is James Spivey. I work with This Dot and we have teamed up with Applitools to offer free visual testing to great open source projects such as this one. Even better, they want to support helping women juniors through an apprenticeship program to do this work. This Dot would handle the pull request for this issue through the Apprenticeship program.

If you're not familiar with visual regression testing there's a pretty good summary in the Storybook docs (most of the page is unrelated to Storybook) but the tl;dr it involves rendering components individually in the DOM and taking a snapshot of the pixels and doing comparisons of it against baselines.

Most solutions do normal 1:1 pixel comparing, but this often results in false positives because the exact pixels can vary. e.g. different GPUs render things differently, so your local tests will fail against baselines created on another computer, or if the CI changes its GPU or there's a browser update, etc, etc, etc. Applitools on the other hand tries to visually compare like a human would, ignoring differences that a human wouldn't perceive or care about. The whole AI thing and such.

Maintaining a separate set of visual tests can could be a pain so what I am thinking instead was take advantage of the wonderful existing docs setup this project has. Since this project already has the ability to render the component examples all by themselves e.g.https://material.preactjs.com/component/button/ this could make rendering and testing the components easy.

My suggested solution would be to integrate Cypress and Applitools to load the docs, render the examples, and snapshot test the dom elements created from the docs.

Looking forward to talking further about this idea and the value it could add to Preact Material!

[cromefire]

CC: @prateekbh

[ladyleet]

+1 and happy to answer questions! :)

[prateekbh]

Hi @myspivey I checked out applitools and it looks good 👍 We currently have pixel to pixel matching using puppeteer but it not the most amazing thing that we like to maintain.

So any replacement needs to have following things

1. Cross browser testing on already chrome and Firefox (edge would be good to have)
2. A better way to see diff between images(currently we compare the pics Manually)
3. Testing on smashed viewport to emulate phones

Does those get covered by your suggested setup?

P.S. Please keep all your work based on next branch instead of master, as that's where we are developing code for our next major release

[prateekbh]

Also can we add support for doing interactions on screen and comparing results post that?

E.g. Load page => do a swipe => compare screenshot to check if the sidebar opened

[cromefire]

Cross browser testing on already chrome and Firefox (edge would be good to have)

Edge (Trident) will be chromium soon anyway leaving Blink and Gecko the only mature engines (besides Webkit, but that's buggy like hell)

P.S. Please keep all your work based on next branch instead of master, as that's where we are developing code for our next major release

The branch is called 2.0 not next

[myspivey]

@prateekbh Pixel to pixel is quite frail which is one of the advantages of Applitools.

1. Chrome and Firefox are both fully supported
2. This process is handled in the Applitools UI where you can see baselines and new images as well as a map of what the perceived changed are. It is a great UI for this! It even handles the ability to approve or deny changes.
3. You can set any viewport size you would like in the config or browsers so if you would like to test chrome or firefox at multiple different resolutions, Applitools will handle this for you.
4. For interactions, this would be E2E testing so a proposed solution of Cypress tied to Applitools would be ideal for this. With Cypress you can interact with your UI as you would expect, and then take a screenshot at any point during to determine if the UI looks as though you would expect. You can of course do this solely in Cypress as well by determining if the dom elements are present after the click, it is at your discretion if you want to have a visual result test or simply know it worked.

Let me know if there are anymore questions!

[cromefire]

4. For interactions, this would be E2E testing so a proposed solution of Cypress tied to Applitools would be ideal for this. With Cypress you can interact with your UI as you would expect, and then take a screenshot at any point during to determine if the UI looks as though you would expect. You can of course do this solely in Cypress as well by determining if the dom elements are present after the click, it is at your discretion if you want to have a visual result test or simply know it worked.

At Applitools it is also stated that selenium works

[prateekbh]

If Cyprus can do the job cross browser I'd like it to, instead of selenium. Cyprus definitely seems nicer, one dependency that can do a bunch 🙂

[cromefire]

I just had a quick look at it and I think we should discuss this later when we have tried some things

[prateekbh]

Like?

[prateekbh]

I am kinda super eager to get this in because as we add components in 2.0, I don't want to keep uploading images to my PR. That too when we are getting an external contributor to do just this. Also, this gives us stuff which we don't have right now.

I honestly think I'd want to land this PR as soon we can after it is submitted.

[cromefire]

I think we should land this first and then think about Cypress vs. Selenium and co.

[prateekbh]

Oh ok fair enough

[prateekbh]

@myspivey we'd just like screenshot tests without Cypress for now, and we'll add whatever we need later on.

[myspivey]

@prateekbh understood, however I would advise this will make it substantially harder and even potentially lose a bunch of the advanced value you can get out of Applitools. Without an E2E runner to help pluck the dom elements, you are using the pure image comparison approach which could lead to false positives/negatives. When instead of sending images, you send dom elements to Applitools, they will ensure they get rendered in a consistent manner so that you do not get differences between different contributor machines. If you do not end up with that runner, the only choices will be the CLI or pure JS libraries that deal simply in images. You would continue to have to ensure that something takes accurate photos per browser, and then deal with upload. You can find documentation on those solutions here (https://applitools.com/tutorials/screenshots-javascript.html#run-your-first-test) and here (https://applitools.com/tutorials/screenshots-cli.html#steps-for-running-tests)

[cromefire]

@myspivey I mean if we start with Cypress change it in the end (if we really want) right?

[myspivey]

@cromefire yes, it is a very lightweight approach. You can see an example of adding Cypress together with Applitools here (https://github.com/auth0/lock/pull/1597/files)

[cromefire]

Well then I don't think there's anything speaking against it

[cromefire]

This is now implemented