Screenshot testing via happo.io

[trotzig]

👋Hi there!

I've been playing around with a screenshot test suite for material-ui. For those of you not familiar with screenshot testing (sometimes referred to as "visual regression testing"), it is the process of taking screenshots of a UI before and after a change to catch bugs and visualize change.

Happo.io is a cloud service for screenshot testing. It is the continuation of Happo the open source project, which I and @lencioni founded in 2014. Happo.io recently added a free plan for open source, and I've been looking for good projects to integrate with lately. Material-UI is great in the sense that most things are already in place: there are examples that are displayed in the docs that we can use when running screenshot tests.

If you want to get a better understanding of what an integration with Happo would look like, check out my local fork where I've been testing things: https://github.com/trotzig/material-ui/tree/happo

Here's an example of a report created by happo.io for Material-UI: https://happo.io/a/31/report/dev-6e9b6986af4b7e986fc2

Whenever a diff is found, Happo will provide a link to a comparison page. Here's one I created by forcing a bug in the CustomizedSnackbars example: https://happo.io/a/31/compare/dev-5b74a5353d936caa003c/dev-7453bf186ff38e6cccdb

To fully integrate Happo with this repo, I would need the following from someone with admin rights:

• Install the Happo Github App in this repo. With the app installed, Happo will be able to post statuses back to PRs/commits.
• Provide me with an email address that I can add as an admin to the Happo account for Material UI

Let me know what you think! I'm happy to answer questions here, over email or in the happo slack channel.

[lencioni]

@oliviertassinari Does this sound like something you'd be interested in taking for a spin for a while? I think it could be really useful for this project!

Note that the report that Henric linked to is just an initial screenshot report. Once Happo is set up, it will only report on visual diffs caused by the PR.

[oliviertassinari]

@trotzig @lencioni 👋 Thanks for sharing the link! We have been developing a screenshot diff service under https://www.argos-ci.com/. We even have one integration example with Happo.

We are already using it for Material-UI (1 year+) for component testing and at the office for full page testing. We have no issue with it so far. It's working great. https://percy.io/ Is another tool I have been following closely.

[trotzig]

Ah, that’s good to know! I’ll continue to use material-ui locally — it’s a good benchmark. And let me know if you want to discuss general screenshot testing strategies, I’m always interested in learning and sharing with others. 😊

[lencioni]

It looks like you are using the open source happo project. happo.io has made a number of improvements on top of what is available there, so it still might be worth trying out.

[trotzig]

^ this is true! The main highlights are:

• support for more browsers (including IE)
• faster builds (the screenshooting phase of material-ui takes less than 30 seconds)

[oliviertassinari]

@trotzig Support for more browsers is interesting, but from my experience, a faster build is more valuable. I have seen very few rendering issues between browsers, mostly with IE11.

[trotzig]

I agree that rapid feedback is vital. That and preventing spurious diffs (false-positives). It’s nice to be able to run through everything in IE every once in a while though (every deploy for instance).

[oliviertassinari]

Wow, I have just discovered some new alternatives with https://www.chromaticqa.com/: