Investigate Visual Regression Testing

[arschmitz]

Look into visual regression testing using image comparison

For anyone looking for the unit testing issue that was moved to #35

[sfrisk]

Has anyone looked into this at all?

[jzaefferer]

I just looked at the two references. Apparently you write "tests" sass in both cases, so that you can test mathematical functions implemented in sass, where the test provides input variables and compares the output to other variables. Based on that these are only useful if this framework ends up with lots of non-trivial calculations that can be abstracted and tested separately. Otherwise, I guess, the declarative nature of CSS leaves little room for unit testing.

Maybe a better idea is to look what kind of testing other CSS frameworks are doing. We started with a list of frameworks, but I can't find that anymore.

[arschmitz]

@jzaefferer @scottgonzalez and I have a list in a google doc of everyone we attempted to contact if thats what you mean? or there were a few lists in the old repo related to specific things. I can get any of these lists still.

[jzaefferer]

I'm just looking for a list of source repositories of various frameworks to see what testing they do.

[arschmitz]

I can do that ill post shortly

[arschmitz]

@kristoferjoseph do you have any experience with this? I know you did a lot of testing related work on topcoat.

Also this seems like it may tie into #23 as a way to ensure the things we claim in our design language are actually true after all pre-processing.

[kristoferjoseph]

Yes we did a lot of investigations into the unit testing of CSS.

After trying everything we could come up with we ended up doing very simple css file comparison tests. I would generate CSS then hand tune it until it was what we needed then used that as the expected output for the preprocessor. These test allowed us to factor out utility classes and know the output was still what we expected. It did become a chore to keep these tests running though. For instance when a bug was found that needed to be fixed in a utility we ended up having to update all components that used that utility, which is ideally what you want, just a lot of work. I always felt there was a way to automate these test updates, but never had the time to create it.

We looked into image comparison, but found that those tests were incredibly brittle and added a lot of maintenance overhead for our tiny team. To me tests like this would only be useful when there is a way to input a design file and export preliminary CSS for refactoring ( Sketch plugin? ). This would then allow you to refactor the CSS and know you were not introducing visual differences, OR better yet you would be aware of subtle differences and be able to decide if you could live with 1px shifts etc.

At the end of the day we felt that the most important tests were around performance. These tests elevated a feedback loop that helped inform our designs.

[markelog]

Do you want to test the SASS code or design representation itself?

If both or the second one then i can come up with couple ways to do the latter.

[arschmitz]

@markelog if i understand correctly 1) is testing that the math within sass is correct, which is what the frameworks i originally mentioned were about and my intention.

• I think would help stick to some of the thinkgs @kristoferjoseph mentioned in #23 like proportions and radius etc... and could be potentially useful though i'm not sure how much of this there will actually be.

2.) Testing if something is visually correct or broken by a change to the css

• This would be super useful in general if there is a good way to do it i was not aware of a good one which I think is what @kristoferjoseph was saying his experience was also. So @markelog if you could come up with something for this that would be AWESOME

[markelog]

I'm talking about second item, specifically third paragraph of the @kristoferjoseph points.

Image comparison with render results and all that, i know at least one example of successful implementation of that technic for the components library.

There is quote a lot of tools for that now, one of them can even produce code coverage data for the CSS styles, excluded areas, smart diffs, cross-browser testing and stuff like that.

It would require you to store in the repo screenshots of the design for every element you have. But it's best way to do automatic testing for this kind of libraries that am aware of.

[arschmitz]

@markelog that sounds really really interesting. Personally id like to see us at least look into this.

[cvrebert]

I'd be very interested to hear what tools others recommend for screenshot regression testing on multiple browsers. https://github.com/webdriverio/webdrivercss looked the most promising when I researched this recently. (Full data)

[kristoferjoseph]

+1 would feel really good to be able to finally put this problem to bed either way.

[markelog]

@cvrebert Wow, that's a nice list you got there.

WebDriverJS is pretty cool, although it has some issues, like: doesn't have "smart" diff or doesn't account for outline or box-shadow properties.

I'm not gonna repeat what you already know, but will mention tool that has escaped from your list, which is pretty similar to WebDriverJS - https://github.com/bem/gemini

Apart from the what WebDriverJS is missing, it also has code coverage, exclusion feature, has GUI (which, as far is i can tell, is not same as https://github.com/webdriverio/WebDriverJS-adminpanel), has a "smart" diff (see https://github.com/SevInf/looks-same and https://github.com/bem/gemini for more info).

But, i'm not sure if it has viewport feature, but since this is the same team from https://github.com/jquery/css-chassis/issues/4, i'm pretty sure it could be added if needed (if it's not already exist that is).

I have an implementation example of that tool to the components library - https://github.com/bem/bem-components: config - https://github.com/bem/bem-components/blob/v2/.gemini.yml test example - https://github.com/bem/bem-components/blob/v2/gemini/button.gemini.js screenshots example - https://github.com/bem/bem-components/tree/v2/gemini/screens/button

[sfrisk]

+1 Nice!

[cvrebert]

Apart from the what WebDriverJS is missing, [Gemini] also has code coverage,

Interesting!

exclusion feature,

WebdriverCSS has that too.

has GUI (which, as far is i can tell, is not same as https://github.com/webdriverio/webdrivercss-adminpanel),

Hmm, now I'm curious what the differences are there.

has a "smart" diff (see https://github.com/SevInf/looks-same and https://github.com/bem/gemini for more info).

WebdriverCSS offers integration with a third-party service that's similar.

[jzaefferer]

Btw. I think this is the list @arschmitz mentioned earlier: https://docs.google.com/spreadsheets/d/1dT9NJGi0pYduz2x0w0Y-kAybw-t4FRtJhBGKwAwQrcU/edit#gid=16765613

Good if anyone wants a list of CSS frameworks for any kind of reference.

[arschmitz]

@jzaefferer very close one tab off https://docs.google.com/spreadsheets/d/1dT9NJGi0pYduz2x0w0Y-kAybw-t4FRtJhBGKwAwQrcU/edit#gid=1913465830

[markelog]

@cvrebert Sorry it took me so long to answer, long holidays, then work, then waited for the clarifcation from the team.

Hmm, now I'm curious what the differences are there.

This is web-service in which you would have to load your data, basically, like html-reporter, whereas gemini-gui has goal to have most of features with the corresponded cli tool, basically like git and git-gui.

WebdriverCSS offers integration with a third-party service that's similar.

Sure, but gemini doesn't require you to use third-party app for it, it has this feature right in the box. Can't say which algorithms work better though.

Checked about viewport, yeah, gemini has it.

Did you start working on implementation of any of these tools to your workflow? I think it would be beneficial to have an example of implementation of this kind of testing to the css-framework.

[cvrebert]

@markelog

This is web-service in which you would have to load your data, basically, like html-reporter, whereas gemini-gui has goal to have most of features with the corresponded cli tool, basically like git and git-gui.

I'm ideally looking for something that exposes a persistent web front-end and a CI-like workflow. Basically: generate new screenshots once per day, if there are unexpected differences then notify the team somehow, and team members login to some web app on the server to accept/reject the changed screenshots.

Did you start working on implementation of any of these tools to your workflow?

Nope.

[markelog]

This feels like a bit out of scope, these tests could be executed like any other tests. Whereas notifications, time of execution, etc; sounds like a CI-tool job, whereas these instruments could be integrated to.

[markelog]

Thinking out loud here, i will have free time in a week or so, i could try to implement one of the candidates to bootstrap workflow and then, if it would be successful, we could use it as an example in topcoat and chassis?

[kristoferjoseph]

@markelog sounds good. Feels like the best way forward is to try them out and see if the results are good enough to go with.

[markelog]

@cvrebert Thoughts on my lost comment?

[cvrebert]

@markelog By "implement one of the candidates to bootstrap workflow", do you mean setting up Gemini for Bootstrap? As a separate repo or something?

[markelog]

By "implement one of the candidates to bootstrap workflow", do you mean setting up Gemini for Bootstrap?

Yeah, fork, then PR - the usual stuff.

[cvrebert]

I can't speak officially for the rest of the Bootstrap Team (heck, I only observe css-chassis in a personal capacity), but I suspect you might potentially get some pushback regarding storing the screenshots in the main repo (which is how I understand Gemini normally works).

[markelog]

Do you have an issue about it? I guess it might beneficial for chassis team to keep track of your progress on it.

[cvrebert]

https://github.com/twbs/bootstrap/issues/11410

[sfrisk]

@markelog Sounds interesting. I look forward to seeing the results of your attempt at getting Gemini working with Bootstrap.

[sfrisk]

Also renaming this, since this issue seems to be more about visual regression testing rather than unit testing frameworks for SASS.

[markelog]

Didn't get response from @mdo in relevant ticket yet, since they consider it as "nice to have thing" i'm not sure of their level of interest.

Will see how it goes.