Add tests

[trezy]

Testing visual frameworks is haaaaaard, but we can do eet! I prefer BackstopJS, but any visual regression testing tool would work. This would go very well with #38, as we’ll need something that renders the pieces of the framework to test. Adding docs would also go hand-in-hand with this as we could then just run the tests against the docs.

[trezy]

I'm going to start working on something for this today. Here's what I'm thinking so far:

Setup Storybook

Using Storybook, we can easily create reliable, testable pages with configuration options for each component.

Setup BackstopJS

Using Backstop, we can create regression tests for each component. Testing obviously provides a lot of value, but the biggest win here is that we can disallow PRs from being merged if they cause regressions.

Setup Storybook Deployer

Storybook Deployer handles deploying storybooks as static sites. this would allow us to actually host our Storybook on Github Pages for users to fiddle with. 😁

[abdallahalsamman]

I'm implementing BackportJS, I'm afraid I'll have 100s of scenarios to test all storybook's components, and for every new component or styling we should add a scenario for a new storybook story or variant.

I was thinking if we could render all stories and their variants on one page, and test just that one page, so when any new story or variant is added, its automatically tested by BackportJS.

I see there was some effort put to create functionality to load all stories and their variants on one page.

Is this a good/right way to solve implement BackportJS with Storybook?

[trezy]

First, let me verify that I'm understanding your concern. I believe that you're referring to this section of the Backstop config, and your concern is that adding scenarios for every variant of every component to that one file is gonna suck. I would agree with that, but I don't think that putting all of the components with their variants on one page is the right solution. It'll make it much more difficult to determine where exactly the issues are occurring.

Alternatively, we could switch to Storyshots. Basically, it's a Storybook addon that will automatically add snapshot testing in for each of our stories. We may need to write additional tests to make sure it tests each variation of each component, but it looks like it'd be a pretty great starting point.

[trezy]

This article indicates that we can pass a --configPath flag to Backstop. With that flag, we can point to a *.js file instead of the default *.json file. This would allow us to generate a better set of test scenarios, and do so more dynamically. This would eliminate the need to write a new test for every variation of every component. Instead, we can create tests for each component that list the various state options, then generate the full list of potential variations that gets passed into Backstop.

We could potentially go even further, creating our own version of Storyshots that leverages Backstop, generating all of our URLs to test from our stories and knobs. That's probably a bit overkill at this point, but the option is there.

[abdallahalsamman]

I don't know if we will have to create our own version of Storyshots, that would be a fun challenge to work on. However, the proposed solution should work well for our current situation.

@trezy nice work on that research 🙌

[trezy]

Hey @evexoio, what's the status on the Backstop work?

[abdallahalsamman]

@trezy hey, I've done the tests where the scenarios are just radios(e.g. balloons, buttons) in storybook.

Now I'm figuring out a way to generate all the scenarios when using radios, select and boolean scenario types(e.g. containers, icons).

I'm getting there 🙃

[trezy]

No worries, @evexoio, thanks for the update! I'm excited to get the tests done so we can prevent regressions via CI. 😉

[montezume]

One thing that might be worth looking into is using a service like percy to run the tests. That way the tests themselves don't need to live in the repository, and all of the issues with rendering differently on different machines don't have to be dealt with.

I recently integrated visual regression testing for the ui component library for which I'm responsible at work, and a combination of jest, jest-puppeteer, puppeteer and percy worked quite nicely. If you're curious, I wrote a blog post about the experience here. I know that percy does have some integrations with storybook (we didn't use it though), and they do have a free tier of 5000 snapshots per month.

We combined multiple states of the components into one snapshot per component to save on snapshot $$$. It should be doable to use a free plan with this repo.