🐈 Visual regression testing

[asvae]

vue-book doesn't provide any means for automated testing, which is crucial for larger apps.

I can see 2 directions to go: 1) Implement vue-book functionality on top of storybook, allowing you to add all demo files. Which means we can use storybook recognition solutions. 2) Introduce all functionality in 'vue-book'.

References

• Storybook - automated visual testing

Initial discussion

• 24

[davidmoshal]

Storybook has always disappointed me, last I looked you couldn't use Vue Devtools with it because of the iFrame issue, so that was a show-stopper. The other issue is their 'knobs' and other tools - why introduced these when you have all you need with Vue? Bottomline, Storybook too heavy-weight for me, and requires too much effort devoted just to making it work, and just isn't worth the effort. Vue-book, on the other hand works just fine with one file and less than 10 lines of code!

So, I'm going to look at the libraries in the issue referenced above, and also see what it would take to integrate with one of these tools, from the Storybook link above:

Applitools Chromatic Loki Percy Screener jest-image-snapshot

Given that these tools just need a list of urls, I'm not imagining it will be a huge effort.

Goals for me would be (in no particular order):

Given a string to filter against the components

a) ability to capture a static html site with pictures. b) ability to capture a pdf with pictures. c) ability to run headless on server. d) ability to run in different browsers. e) ability to run with different resolution sizes. f) ability to run with different Vuex states. g) ability to run with different themes (for me these can be data driven, eg using Vuex). h) ability to integrate with a visual diffing service ie from the above liste

[davidmoshal]

OK, so here are my initial findings:

a) Many of the libraries in my initial note use PhantomJS which is not being actively developed, and probably isn't relevant to any real-world browser. Therefore PhantomJS-based libraries should be avoided.

b) Regarding the cloud-based solutions listed above:

• these are expensive, only some offer a free tier, but typically this is only 120 minutes per month.
• percy.io seems to have the most affordable paid service at $29/month for the first tier.
  • it has Vue integration, but only via Storybook
  • however, it has other integrations which would work fine, e.g. cypress.io integration, which seems very straight forward.

c) I tried chrome-pdf and was very impressed.

• It uses puppeteer (Chrome), and worked right out the box first time!
• It can generate pdfs and images no problem.
• The installation lists a large number of required libraries, but my Ubuntu 18.04 had all of them already.
• Of course this is Chrome-only so can't be used for cross-browser regressions.
• However it completely meets my needs for testing themes, generating portfolios, and for visual site backups.
• It's also probably linux only.

d) For cross-browser testing, this library looks great (I've not tested it yet): html2canvas.

• It works cross-browser.
• For some browsers it would probably need to be triggered inside the actual browser I think, unless there is a way to automate e.g. IE ?

e) For visual regression, this library looks promising: gemini.

• However it requires Saucelabs or Browserstack, which are both expensive.

f) If you have images from different browsers (eg using html2canvas) then this library can be used to do regression tests: jest-image-snapshot. It uses pixelmatch.

Conclusions:

1) for Chrome-only pdf and image generation on Linux, then chrome-pdf is excellent.

2) for multi-browser regression testing then I think html2canvas is worth looking at.

• as mentioned above It would need to be manually triggered inside some browsers I think, but thereafter could be used to automatically capture images, generate a static site, and, using say jest-image-snapshot could create pixel-level regressions.

3) For completely automated, cloud-based, paid, service, which is likely to exceed 120 minutes per month of testing, then Percy.io + cypress.io seems like an option to explore.

I'll take a look at #2 and #3 above and report back.

[asvae]

Great! That would be very helpful.

I checked a couple of things from my side.

We can already get a list of urls:

const VueBook = require('vue-book').default

const demos = VueBook(require.context('./..', true, /.demo.vue$/), '/demo')
const paths = demos.meta.demoFileCollection.demoFiles.map(
  demoFile => demoFile.path
)
console.log('paths', paths)

We will probably need some kind of flag to hide navigation panel, which I'll look into.

Thanks for investigating. vue-book is not entirely on my radars right now (no business requirements pending). So tackling that on my own would take a while.

[davidmoshal]

Thanks, I'll try that shortly.

I'm thinking that the easiest way to integrate with the various tools is:

• Create route on the browser side which returns a list of urls. eg http://localhost:8080/#/demo Filtering could be handled there /demo?filter=... or on the returned urls.

• Then the tools (chrome-pdf, html2canvas, cypress.io) can first query that url, and iterate through the list of urls, and generate the images, or call out to percy.io etc.

I think this approach would work fine with chrome-pdf and percy+cypress, and avoids having to run anything specific on the server side, ie: the tools can be in another process entirely.

Not sure about multi-browser testing with html2canvas, but honestly if the above 2 work then that's fine for my requirements.

Dave

[davidmoshal]

found a great resource: https://github.com/mojoaxel/awesome-regression-testing

[asvae]

Added a flag to make navigation panel disappear.

It's present in version 0.1.0-alpha.6.

Usage example:

import { createRoute } from 'vue-book'

const router = new Router({
  routes: [
    createRoute({
      requireContext: require.context('./tree', true, /.vue$/),
      path: '/demo',
      hideFileExtensions: true,
      hideNavigation: true,
    }),
  ],
})

I'm thinking that the easiest way to integrate with the various tools is

We can probably just push these urls to a file to make it accessible from shell.

Filtering could be handled there

You can use regex here: require.context('./tree', true, /.vue$/) Or is regex not enough?

[davidmoshal]

We can probably just push these urls to a file to make it accessible from shell.

Thanks, that will work for local development but for testing when the browser is hosted remotely, e.g. for automated cross-browser screenshots on a hosted service, then that won't work.

That's why I think it would be easier to just query the browser for the list of urls, and then use that list for the screenshots.

Similarly, it would be nice to be able to hide the panel dynamically based on an url switch, because if the browser is hosted remotely, we can only control it via the url.

I'll try a proof-of-concept of both of these.

Did you get the email I sent you? Dave

[asvae]

That's why I think it would be easier to just query the browser for the list of urls, and then use that list for the screenshots.

So are you talking about static url here or component is ok? (so whatever queries the browser would have to wait until everything is loaded)

panel dynamically based on an url switch

Not a problem.

Did you get the email I sent you?

Nope. Checked spam, still nope. Added email to my github profile. You also have me added in discord. Tomorrow I'm mostly free, so chat or call is possible.

[davidmoshal]

I resent the email to the new address, let me know what you think.

[davidmoshal]

So are you talking about static url here or component is ok?

Either one, the idea is to send the browser an url, eg:

  http://localhost:8080/#/vuebook/urls?query=*

(so whatever queries the browser would have to wait until everything is loaded)

Yes, but really only the panel needs to be loaded (mounted).

Then we get the list of urls (this could be json, or even just<pre>), ie:

<pre>
  http://localhost:8080/#/book/apps/book/Book.demo.vue \n
  http://localhost:8080/#/book/components/users-table/UserTable.demo.vue \n
  http://localhost:8080/#/book/pages/Page.demo.vue \n
</pre>

Then we can use each url for a screenshot: e.g:

  http://localhost:8080/#/book/apps/book/Book.demo.vue?panel=false
  http://localhost:8080/#/book/components/users-table/UserTable.demo.vue?panel=false
  http://localhost:8080/#/book/pages/Page.demo.vue?panel=false

The query string is running in the browser btw, not the server. So, that way, we can control the browser without touching the backend or file system at all!

[asvae]

So, if we have access to compiled page - we can get these routes quite simply.

document.querySelector('.vue-book-root').__vue__.treeFileCollection

I'll answer your email tomorrow if that's ok. Need to finish a little something today. Overall I'm positive. Seems like a small fun project that could potentially grow into something big.

[davidmoshal]

I'll answer your email tomorrow if that's ok. Need to finish a little something today.

Understood, no hurry.

Overall I'm positive. Seems like a small fun project that could potentially grow into something big.

Exactly, and it would be a real help to the community too. (and it would be more useful than, say Applitools, even though they raised $42m)

Dave

[davidmoshal]

So, if we have access to compiled page - we can get these routes quite simply. document.querySelector('.vue-book-root').__vue__.treeFileCollection

Agreed, or if we use a component, then we just hide the whole page and have a component which shows the filtered <pre>list of urls</pre> no? (seeing as we have the power of Vue to help)

[asvae]

Doing proper html is definitely not a problem. My solution just allows to skip a step, exposing js array directly. Given there is no much a point to make it human readable.

[davidmoshal]

Given there is no much a point to make it human readable

true