Proposal: Visual Regression / Screenshot Diffing

RandallKent commented 7 years ago

We probably will not do this as a built-in feature in Cypress just yet. Instead please consider using open source plugins or commercial systems that built on top of Cypress test runner.

Here is our documentation guide: https://on.cypress.io/visual-testing

Find image diffing plugins in https://on.cypress.io/plugins#visual-testing

ezeedijk commented 7 years ago

+1 on this feature request! We're now using blink-diff to compare screenshots generated by Cypress for visual regression testing, but having this integrated would be much better.

rmharrison commented 7 years ago

@ezeedijk: These guys do continuous visual integration (pixel perfect), but not functional BDD/TDD: https://percy.io/

Pro: Have out of the box plugins for popular MVC frameworks, e.g. ember-percy Con: No free tier. Starts at $150/month after 2-week trial.

Seems like visual regression would go along great with Cypress, esp. given the dashboard. Functional test + visual diff all in one dashboard.

rndmerle commented 7 years ago

@ezeedijk , if you don't mind I've some questions. Are you using blink-diff with an assertion right into a Cypress test? How are you handling the validation? (updating the reference screenshot)

leonardoanalista commented 7 years ago

Image diff would be a great feature and make cypress a solid alternative to phantomCSS/Casper for css regression testing. Please ensure we can compare portions of the screen and exclude page sections too.

ezeedijk commented 7 years ago

@rndmerle No we're using blink-diff in our CI after we've run our Cypress tests. We've simply created three folders:

'screenshots' - new Cypress output
'screenshots_base' - set of correct screens for blink-diff to compare against
'screenshots_diff' - where blink-diff outputs the files in case of error

Updating the base set of screenshots in case of changes is a manual step. We use it very limited just to discover changes in parts of our application that are very difficult to assert other than visually, in our case mainly graphs. We do still have stability issues in some tests due to the a-synchronise nature of the cy.screenshots. So it would be nice to have visual assertions directly into Cypress.

leonardoanalista commented 7 years ago

@ezeedijk can you take a screenshot from part of the screen like phantomCSS? Example, from body or #contents.

brian-mann commented 7 years ago

Yes. We control the DOM. We can basically do anything we want prior to taking the screenshot. It would be trivial to "blackout" all the other areas, or even calculate the box model for the thing you want to screenshot and then resize the image accordingly.

eskitek commented 7 years ago

We currently use PhantomCSS for visual regression testing. We use it in combination with PhantomFlow, which provides very nice visual diff presentation and the ability to replace baseline images easily.

Usage: We take targeted screenshots at various points during our functional test runs. We don't use screenshots to replace functional assertions per se, but we do take them within functional test flows to verify the visual state of sections of the page under various conditions. Or goal is to screenshot every screen element once under each of its states. Some functional tests may have several screenshots taken within them, others may have none.

Overall, we like the functionality of PhantomCSS a lot, but we are looking for a replacement that doesn't require PhantomJS. We are excited to see what Cypress will do in this space. Some things we'd like to see:

Ability to screenshot different parts of the screen by selector (xpath, ID, etc)
Ability to take screenshots during a test run at whatever point we choose to
Ability to easily visualize which pixels don't match between 2 images
Ability to replace the baseline image easily (Example: PhantomFlow provides a "rebase" button. See https://github.com/Huddle/PhantomFlow#rebasing-visual-tests)
Ability to set threshold for percentage mismatch that will trigger a failure

ManualTester commented 6 years ago

.

garris commented 6 years ago

Hi guys, I am the maintainer of BackstopJS. I wonder if there is some kind of interesting integration that could happen here? Happy to chat if there is some interest.

bahmutov commented 6 years ago

@ezeedijk we have a proof of concept using blink-diff and that is very nice workflow - and with some UX love could be game changer. Stay tuned!

nikosgpet commented 6 years ago

+1 on this feature request!

zth commented 6 years ago

@bahmutov I was very intrigued by your talk of your concept - any update on that? I'm looking to do the same thing basically.

wnvuong commented 6 years ago

+1

rsudarson commented 6 years ago

It would be really nice if cypress supports this feature sooner.

mikkoronkkomaki commented 6 years ago

👍

arcticlinux commented 6 years ago

+1

maxime1992 commented 6 years ago

Me when I receive plenty of +1 notifications in 2018

.	.

This is not helping guys, just disturbing for the whole discussion. Use reactions... ... add something usefull ... or make a PR :sweat_smile:

I mean, if the topic was dying I might understand, trying to get attention.

But with Gleb saying:

@ezeedijk we have a proof of concept using blink-diff and that is very nice workflow - and with some UX love could be game changer. Stay tuned!

It does not seem that we're being abandoned :wink:

julianburr commented 6 years ago

hey @bahmutov, was just wondering since you seem to be actively working on this feature but the help wanted label is still on, if there is anything we could help with? :blush:

Fwiw I quickly threw something together (https://github.com/julianburr/cypress-match-screenshot) cause I needed to do some screen diffing for a project, but that’s obv just a temporary solution. Really looking forward to proper official support for matching screenshots.

Btw, thanks so much for your work in general. Been working with Cypress for the past couple of months and its been amazing :tada::pray:

bahmutov commented 6 years ago

@julianburr it looks good, we will discuss this as soon as I get a little bit of time - we made the same thing, the pixel comparison is easy, the harder part is coming up with the entire workflow. Like where do you store images, how do you approve changes, etc.

egucciar commented 6 years ago

@bahmutov if you release an API for it let us know. I need to adjust what @julianburr did to work in CI and not sure if we have the time for it.

seanpoulter commented 6 years ago

I'm really interested in this issue. How can I help?

jennifer-shehane commented 6 years ago

We will soon be releasing some steps we've taken to get closer to screenshot diffing as a feature. The main issue is covered here: https://github.com/cypress-io/cypress/issues/1424

Keep an eye on the changelog for full details.

khrome83 commented 6 years ago

@jennifer-shehane - we are looking to implement the solution mentioned above with external diff tool. We would love to know if we are "close" as in days away with this support within Cypress, or if this is much further out. I just read the entire Change Log and I can see the changes made moving towards this support.

We are comfortable enabling the 3rd party solution if we have a few weeks or more to wait. I just need a general idea of when we can expect this to drop in a release.

brian-mann commented 6 years ago

@khrome83 Really there's nothing we have to do in Cypress as of today because we generate the screenshots on the filesystem, you can do whatever you want with them now.

However it would be nicer if we added an after:screenshot event in the pluginsFile so you could use node immediately there to do whatever it is you want to do (such as diff) the screenshot.

egucciar commented 6 years ago

@brian-mann

I think he's asking if Cypress will implement the fully featured diffing ability within the next few weeks and if not it's worth his investment to implement his own based off the cypress screenshots.

I have implemented our own version but need to swap the diffing algorithm because the one I leveraged is not sizing images correctly. From what I understand full featured diffing is not on the immediate roadmap so it's not to be expected to be shipped anytime soon

brian-mann commented 6 years ago

@egucciar we have a clear idea of what all we want to do, but right - it's not going to come within weeks, that timescale is more about months.

Doing the actual diffing computations on the screenshots on disk is the easiest part that you can implement in a single day. We've already done the heavy lifting to take an accurate screenshot, full page, element, etc. However, adding the automatic ability to retry the screenshot when it doesn't match is more complicated and would more work to do.

It gets really complicated actually building a process / service around a fully baked screenshot diffing system. Diffing a homogenous environment is easy - but involving CI, "reviewing the screenshots that don't match", approving / denying changes, etc is substantially more complex. That's where the service / Github Integration / Status Checks part comes in that we care about.

egucciar commented 6 years ago

Right. I completely understand this and therefore I recommend that the community, if needed, implements their own version. Unfortunately since the diffing algorithm integration with node is not as trivial as you make it seem, I haven't even gotten that far past just being able to call my own node code from cypress and therefore cannot share my code since the diffing isn't working yet. (It diffs, but the diffs fail due to image sizing issues) But at least I'm able to integrate node code with Cypress code (boy is it ugly though, calling cy.exec(node <direct path to node module which executes the node part of the code>)>. Despite being ugly , at least it works but I forked cypress-match-screenshot and it tries to be smart by resizing images and it's regressed with Cypress 3.0 and I want to completely swap out blink diff with resemble.js

It's currently a ticket in my backlog so I won't get to it for a few days but fully intend of having it working within a week or two

brian-mann commented 6 years ago

You can completely nuke cy.exec and replace it with cy.task. That gives you the full power to do whatever it is you want to do in node without needing to smoosh it all into a single cy.exec command.

As long as you return a promise it will be awaited correctly and you can do whatever it is you want to do in there.

You don't need cypress-match-screenshot anymore - our native integration in 3.0 is way more comprehensive and bypasses all of the scaling issues.

I know you're still waiting for 3.0.2 with fixes for cy.screenshot like the DPI issue. It's already fixed and waiting patiently in develop branch.

+1 to resemble.js, in my cursory experience it was the best to use in order to handle aliasing issues when DPI's are different.

brian-mann commented 6 years ago

Also I didn't mean to trivialize the integration - I meant for us as a team to add in an algorithm, expose a plugin event, and essentially "do diffs" is very easy since its all off the shelf and we've done the hard stuff to get it in a position to do that.

As I mentioned, baking this into Cypress officially and automatically designing this into the API's to do diffs and add retrying support takes more effort and thought, but still very do-able.

The other stuff that makes it "all glue together" into a seamless process with CI and PR / Status Check flows is a whole other equation.

Keeping the masters in source control is the easier way to start, but it completely falls over in CI because the environments are no longer homogenous, and you really don't want screenshot diffs to be causing failures in CI. Unlike functional tests (which should never break) screenshot diffs will break all the time due to natural changes in your application, which are not indicate of a bug.

Because of this, the process is very different than functional CI tests. In functional you always want the build to break when tests fail. With screeshots, you don't want them to fail - you want to be able to "review the differences" and approve them if they're supposed to be breaking - all without jamming up the CI run.

The other problem with source control is that you'd need to re-commit those changes to source which would then kick off another CI build. Sure, there's ways around it, but that's really the whole entire problem. Screenshot diffing requires a different process which does not work like how we run functional tests. It's the perfect seam for adding a lightweight service component (which is what we intend to build). However, Cypress will be flexible enough to still do the diffing locally and stay out of your way and let you create your own process if you choose not to elect into that service.

khrome83 commented 6 years ago

@brian-mann & @egucciar - thank you for the meaningful feedback and conversation.

I plan to add a cypress plugin, just because I want to simplify the screen resize and capture management, but I agree that the cypress-match-screenshots is not needed. I will also add a reporter plugin to send the data to a slackbot (or any webhook). From there the team can grant yes/no through slack channel dedicated to this. But like you said above, we won't break the build, as they could all be correct. We will control this in a lower environment, and use process to ensure we do this before production.

I will open source what I can, specifically the cypress plugins. I read the feedback above and will look to use resemble.js over blink-diff as well.

High Level Flow - everything below dotted line is internal to us, but I think speaks to a fairly ideal flow that can be reproduced.

egucciar commented 6 years ago

@khrome83 If desired, I can definitely help out building this, I have gotten far with the visual diffing algorithm which is working really well (full page stitching issues aside), using blink-diff. The entire storing of bases/diffs/etc are all present. There is no issue with screenshot cropping as of current. The slackbot integration would be an amazing piece. I know its been a month but somehow I missed this before. Let me know how you're doing and if you'd like to share any progress/ideas/issues.

awentzel commented 6 years ago

Is there an ETA on when this feature is likely to be available?

We're currently using Jest/Enzyme/Cypress and would love to finalize out visual regression testing with this feature rather than including another stack like Applitools, for example.

btw: really appreciate the thoughtful contributions in the later comments. Cypress is really high quality with outstanding documentation so I'll happily be patient while you finalize and fine-tune the details.

mjhea0 commented 6 years ago

@khrome83 I implemented almost exactly the workflow from your screenshot.

I built a basic plugin for how I handled the screenshot diff'ing here. It's different in production, but the diff logic is the same.

I may have time next month to open source the slack bot. It's permission based - e.g., the number of failures determines who has to approve the overall test run. Instead of forcing people to approve for each test failure, I went with a blanket approval for the entire build. In other words, if approved - all base images are updated.

Slack + Lambda + S3

Did you end up open sourcing your implementation? Would love to collaborate on this, if not.

khrome83 commented 6 years ago

@mjhea0 - this looks amazing. No we got a invite for percy.io and have been using that for the last month. We are beta testing there GitLab integration.

We had some pretty high visibility items come through my shop, so i have not be able to have anyone work on this, and i have not had time either. We got fairly far with a plugin that generates resemble.js comparisons, but your code is much cleaner.

We would love to participate and play around with what you build and contribute back though. I know @egucciar also was looking at this as well.

This is where I left off. https://github.com/khrome83/cypress-visual-diff

Maybe there is something valuable to steal?

We focused on taking screenshots based on configuration, including different breakpoints.

mjhea0 commented 6 years ago

@khrome83 Yeah, I got the same invite from percy. Kind of wish I had jumped on that since we've been just recreating the wheel.

I'll take a look at your implementation later today.

Anyway, the next thing to focus on is how to get feedback to, well, everyone - so we're building a nice dashboard that shows the entire workflow. I would love to open source this as well. Maybe you and someone on your team would like to hack through it over a weekend?

khrome83 commented 6 years ago

Yeah, we would be open to that. I will recommend it to them if you open source it. I think a lot of people are interested in just participating in open-source in general. This is a problem that is directly in-front of them and solves some big issues for front-end development.

How do you find image-diff. We tried blink-diff but was not happy. I know some others found success with it. Ressemble.js seems nice, but uses canvas which limits where/how we can run it. (no lambda, have some issues filled).

mjhea0 commented 6 years ago

Would be fun to re-write the Cypress CI Messaging Approval Tool with Slack, API Gatway, Lambda, and S3.

Just something basic:

Run tests on Travis CI
Pipe out tests results to S3
If any image regressions, package up image links (from S3) and send to Slack.
If manager approves, ping API Gateway and trigger a lambda to deploy

Want to sync up over email?

I came across image-diff when I was spiking on https://github.com/mjhea0/testcafe-visual-regression. Solid project even though it's deprecated.

egucciar commented 6 years ago

@mjhead sounds good to sync up. I think the approach should be flexible to say circleci Integration as well, if possible

pateketrueke commented 6 years ago

I did this few hours ago, glad to share: https://github.com/agave/testcafe-blink-diff

cdeutsch commented 6 years ago

I tried cypress-image-snapshot out today. https://github.com/palmerhq/cypress-image-snapshot

Seems to work as advertised. I'm surprised it hasn't been mentioned here yet

egucciar commented 6 years ago

Definitely going to give it a shot next time, great suggestion.

cdeutsch commented 6 years ago

After trying out cypress-image-snapshot for 2 full days, the biggest challenge isn't having the diffing functionality, it's creating consistent screenshots.

I've tried about 50 different combinations of settings, running in headed and headless modes and I can't get two consecutive runs to produce identical screenshots, even with a 2 second wait before taking a screenshot to make sure animations, etc have settled.

I was previously using Puppeteer and didn't have issues getting consistent screenshots for the same app, so I'm not sure what Cypress is using to create screenshots which leads to inconsistencies.

I have some screenshots, but can't post them yet as the app isn't released. If anybody working on Cypress wants them I can send them directly.

jcdarwin commented 6 years ago

@cdeutsch I've been using cypress-visual-regression and have found it works well -- possibly the best thing you can do regardless of which library you use is to set your difference threshold to something like 5% -- I notice that screenshots can have small differences from time to time such as very slight offsetting of elements, but a threshold of 5% is enough for these not to fail tests. Also, I've found that screenshots will differ from one environment to another, because of issues such as different system fonts etc.

cdeutsch commented 6 years ago

@jcdarwin yeah, if I adjust the threshold tests will pass, but in my experience pixel perfect screenshots are the best way to make sure things haven't changed. There are plenty of unwanted changes that can occur within the range of even 3%, like things slightly shifting, changing color, or losing drop shadows.

With Puppeteer I have no problems creating pixel perfect screenshots reliably. ¯_(ツ)_/¯

ola5 commented 6 years ago

@ezeedijk I read your post and tried using blink-dif in my test but couldn't get it working , could you put me through the configuration please.... thanks in advance

egucciar commented 6 years ago

@cdeutsch i agree that the inability to create pixel-precise screenshots on at LEAST same-runtimes is a huge blocker to us doing this on our own. I have raised a similar issue , but I've only run into it with fullPage captures. I think there shouldn't be much of an issue with regular viewport screenshots & i have had reliable screenshots w/o diffs on same-runtime builds with viewport screenshots. Full page algorithm seems to be non-deterministic creating issues with minor diffs (or even, in some extreme cases, huge diffs), from one run to the next.

I consider diffing to be blocked without cypress teams' support on troubleshooting their own screenshot algos and capabilities & agree that it's an issue specific to cypress due to screenshot diffing being widely usable in other frameworks.

Have you played with fullPage vs viewport screenshots? Have you compared your screenshots on same-architecture runtimes?

this issue is blocking me from fullPage screenshot diffing: https://github.com/cypress-io/cypress/issues/2175

this issue is blocking me from re-using screens taken in open in the run commands: https://github.com/cypress-io/cypress/issues/2102

Finally, I am unsure if I'm able to get reliable results with viewport without a small threshold and agree a small threshold should not be needed - there are plenty of issues which can crop up within a 0.5% threshold, albiet, minor ones. 0.5% is a very small number after all (read ... not 5%!!!)

timhaines commented 6 years ago

Percy has launched visual testing for Cypress today. 🎉 🎉

Details on the blog post.

There's also a very nice Visual Testing with Cypress tutorial if you have a spare 5 minutes. 😄👍

maxime1992 commented 6 years ago

Can Percy work without their own app?

Also from what I can see Percy is definitely not cheap and has currently no plan for open source project.

I really hope that we get a solution baked into Cypress directly :heart_eyes: ...

cdeutsch commented 6 years ago

How much was Percy? I couldn't get their solution to work. All the screenshots were blank.

I tried ApplitoolsEyes and their solution is really nice. Works really well and didn't take too long to setup.

The only issue for me is Applitools doesn't take into consideration scroll position (it's not taking a screenshot, it's saving your DOM and sending it to the cloud to be recreated) https://github.com/applitools/eyes.cypress/issues/9

Unfortunately, Applitools pricing is for enterprises. 5K validations for about $2K per user. 😳 Seems like a lot for a brand new product where issues sit ignored on GitHub for 2+ weeks. 🤷‍♂️

I settled on cypress-image-snapshot with the setup below.

The settings in support/index.js are the tricky part.

At the moment I've settled on customDiffConfig: { threshold: 0.005 } which seems to be the sweet spot in my app for picking up actual problems.

cypress.json

{
  "env": {
    // change to true when you need to update screenshots.
    "updateSnapshots": false
  },
  "video": false,
  // use what works for your app.
  "viewportWidth": 1180,
  "viewportHeight": 720
}

plugins/index.js

const fsExtra = require('cypress/lib/fs');
const glob = require("glob");
const path = require('path');

const { addMatchImageSnapshotPlugin, matchImageSnapshotOptions, matchImageSnapshotResults } = require('cypress-image-snapshot/plugin');

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  addMatchImageSnapshotPlugin(on);

  on('task', {
    // Work around for not being able to call on('task', ...) more then once. 
    // https://github.com/cypress-io/cypress/issues/2284
    'Matching image snapshot': matchImageSnapshotOptions,
    'Recording snapshot results': matchImageSnapshotResults,

    'fileExists': (filename) => {
      // check if file exists.
      return fsExtra.pathExists(path.join(process.cwd(), 'cypress/fixtures', filename));
    },

    'cleanupDiffOutput': () => {
      const snapshotDiffs = path.join(
        process.cwd(),
        '**/__diff_output__',
      );

      glob(snapshotDiffs, {}, function (err, files) {
        for (var xx = 0; xx < files.length; xx+=1) {
          const file = files[xx];
          fsExtra.removeSync(file);
        }
      });

      return false;
    }
  });

  return config;
}

support/index.js

import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';

addMatchImageSnapshotCommand({
  failureThreshold: 0, // threshold for entire image
  failureThresholdType: 'percent', // percent of image or number of pixels
  customDiffConfig: { threshold: 0.005 }, // threshold for each pixel
  capture: 'viewport', // capture viewport in screenshot
});

util/cypress.js

export function screenshot(tag) {
  // Add a delay before all screenshots for animations, etc.
  // 1111 is arbitrary. Just using something unique to distinguish between a screenshot delay and others.
  cy.wait(1111);

  cy.matchImageSnapshot(tag);
}

integration/my-spec.spec.js

context('MySpec', () => {
  before(() => {
    // Cleanup diff output before each run.
    cy.task('cleanupDiffOutput');
  });

});

cypress-io / cypress