Snapshot testing

[joshhornby]

One of my favourite Jest features is Snapshot testing.

We can already test views using elm-html-test but I believe snapshot tests are cleaner and easier to work with.

I know @rtfeldman mentioned on Slack that he is planning on adding snapshots, so this issue is to keep track and discuss ideas

[mgold]

I've used Jest snapshots with React components before, and haven't found them useful. When something doesn't match up, it's not worth the time to review the snapshots to see if there's been a serious regression. When you're maintaining a UI, like Facebook, I can see the use; when you are bringing up a UI and expect DOM changes, they get in the way. My experience is that they cause a lot of headaches and CI false positives, and I've never seen them catch a real bug.

Snapshots (canned output) suffer from the similar problems to fixtures (canned input): over-constrained, inflexible, hard to identify whether a discrepancy is significant or not, and what is supposed to be tested. A good elm-html-test will make sure that data makes it in to the table (for example) while a snapshot test will fail when the table changes styles.

All that said, I am open to being proven wrong. Is there some way to make snapshot tests work without doing too much work, and without rolling them out with the Elm-Test Seal of Approval until we can be confident they catch bugs?

[drathier]

I agree with mgold. Snapshot tests seem to test the code after too many transformations, so it's hard to know what was supposed to be changed and what was a real bug. It also depends on the device, browser, screen size etc. so you have to test a lot of screen sizes and devices, which requires a decent amount of infrastructure. I feel like this is only really useful for larger organizations, due to the high number of false positives. There are tools out there for taking screenshots of websites on a thousand different devices and combinations, and even from different places on Earth, so I think that's a better alternative, but I'm also happy to be proven wrong.

[rtfeldman]

Wow, really interesting perspective!

I've heard positive things from others, but clearly there is another side to this story. 🤔

[joshhornby]

Where I saw the biggest benefit was when refactoring some react components, I wanted the component to look the same way (css styles etc) but instead I was updating some internal logic (in this case how the map and filtering worked). I guess I could achieve the same with elm-html-test but personally I find snapshot testing an easier way to achieve this. As I don't have to write assertions or write multi dom selectors.

There are tools out there for taking screenshots of websites on a thousand different devices and combinations, and even from different places on Earth, so I think that's a better alternative, but I'm also happy to be proven wrong.

I don't think snapshots are equal to taking screenshots, snapshots are used in react land to test small components. Taking screenshots implies needing a browser and then doing some regression testing to compare what has changed, this sounds like it would lead to lots of false outcomes.

When you're maintaining a UI, like Facebook, I can see the use; when you are bringing up a UI and expect DOM changes, they get in the way.

@mgold Can you elaborate on this? I agree whilst initially developing some UI views that snapshots will change and this would be annoying, I personally use them to test against regressions not part of a TDD workflow.

[mgold]

I guess it all comes down to how mature and (relatedly) stable your UI is. It's only possible for a snapshot test to fail if you change a view function. It's only possible for a snapshot test to fail usefully if you are refactoring your view function (not expecting to change the behavior), but make an error not caught by the compiler or some other view test. I suspect that most changes to the view function don't meet those "useful" criteria, so therefore, most snapshot failures are false positives. They detect changes, not regressions. Which brings me back to: what kind of bugs would go uncaught without a snapshot test, that a snapshot test would catch?

[jwoudenberg]

I think the core of the problem here that writing nice tests for your views is hard. It starts with the compiler only giving you minimal guarantees when it comes to your view functions: it will only ensure your view returns Html msg, so basically "syntactically valid Html". The space of "syntactically valid Html" is huge, so the space for errors (returning the wrong valid Html) is huge.

So we write Html tests, but I feel those tend to come with a significant maintenance burden. Writing small unit tests for your views means generating a relatively large amount of Html and then running an assertion on some small aspect of it. Sure, that's relatively precise test that won't easily fail for the wrong reasons, but in my experience when they do fail the error message and debugging experience can be lousy. You get an error like "Expected two occurrences of the classname 'foo' but found none" plus a huge Html string that indeed doesn't contain such a classname and are left wondering what that's supposed to mean. It's unclear exactly what went wrong. Is a list an item shorter than expected? Is the list rendered at all? Does the section of the page get rendered at all? A diff between the broken output and the older correct output is a much better error in that it clearly shows you what is wrong.

I can imagine the amount of pain experienced by false positives will depend on how easy it is to deal with them. If you can 'accept' the new result with the press of a button, I'd happily take clicking that button a couple of times over interpreting Html test failures.

[willisplummer]

I've really enjoyed using a suite of snapshot tests in conjunction with unit tests in Javascript. The unit tests allow for you to assert about the specifics of how a thing will work while the snapshots can catch weirder bugs as the application scales and provide a general picture of what's changing in view components. Definitely think the ability to accept the changes with press of a button is key. Even if it's tempting to just accept the changes without reviewing them, checking in the new snaps is also an opportunity for PR reviewers to catch more discrete bugs and regretions!

[mgold]

If we add snapshot testing, we will definitely incorporate one-button updating and other ergonomic features from Jest. But even that is one button too many if the tests don't catch bugs. Determining whether they do or do not will be much easier with a prototype.

Help wanted: Create a prototype implementation of snapshot testing (likely a standalone script that records the output of Html a as an Elm file). Report back on (1) test writing effort (2) bug finding ability (3) false positive rate, compared to elm-html-test.

[rtfeldman]

Really interesting perspective. These points resonate with me: https://twitter.com/searls/status/919594505938112512

[rtfeldman]

Here's another interesting viewpoint:

we should really call em "snapshot assertions." "snapshots" are just generated fixtures that you make assertions against. they have the same characteristics of all fixtures, but with better DevEx.

https://twitter.com/briantford/status/920349861630717952

[drathier]

I would really like to know if one of the intermediary representations in e.g. chrome would be good to test against, before it gets rendered to pixels. It would be an AST, but with unfamiliar syntax, so you could dig deep and only look at the thing you care about, even if it moved on screen.

[tkreis]

@mgold I've started to hack an simplistic snapshot testing module a while back. https://github.com/tkreis/snapshot/tree/master.

My local version also checks for onClick handlers but this won't work without modifying the elm-html-in-elm, therefore it hasn't been pushed.

[eeue56]

@tkreis Bring it up on Slack in #testing, and I can help out with what you need

[michaeljones]

I find myself being quite keen to use snapshots if possible. Is there any update on this?

[mgold]

As far as I know, no one has taken me up on my October 2nd comment.