Automated Testing: Image Snapshot Tests

[bryphe]

Thinking about #145 - for some of these very visual cases, we have no current test coverage. It's important to be able to make changes safely and confidently - so I always think when there is a regression - how can we improve our 'safety net' to catch these?

We're getting to a level of features with background color, text rendering, borders, shadows that it becomes tough to validate all of these in a PR change!

What I'd like to add to our infrastructure is a set of image-based verification tests, that can validate some of these basic scenarios. These would render a simple scene or component, save it as an image, and then compare that image to a snapshot.

This isn't a new idea; tools like Telerik have supported it for a while. Doing googling shows a RosettaCode problem for an algorithm for this 😄

The challenge with such tests is making sure they are reliable and easy-to-update. For reliability, it often helps to have a threshold (% of pixels with the same value), or use per-platform snapshots (there might be differences in anti-aliasing, for example). These test suites should be pretty limited and focused on the core set of rendering primitives we have, because they have a maintenance cost. But they can help protect us against regressions.

Open questions:

• Is there a library we could use for image-verification tests? Something that could do a bitmap comparison and ideally show some sort of diff (ie, an overlay that shows different pixels in red).
• Which scenarios should we cover?
• When an image-verification test fails, how do we make it easy to act on and know what failed?
• What additional APIs do we need to take a screenshot of the GLFW window, to compare against the snapshot? It looks like the glReadPixels API mentioned in this StackOverflow Post could help.

Alternatives: One alternative to image-based snapshot testing is OpenGL API snapshot testing - essentially, put a proxy in place for all the glXXX calls, that record the inputs. This can ensure we end up with the same set of GL calls. This is more robust then the image-verification approach, but it also is a much higher maintenance cost - any internal refactoring performance improvements that would've passed the image verification test would also flag as a failure for these tests. So I'd lean towards the image verification test, for now.

[OhadRau]

Found this page that might help, although I really know nothing about UI testing. This is paid but it looks like it has a lot of useful features too. Looks like glReadPixels + an image diff could work if we just run each test for ~3 seconds and then take the screenshot. If it fails, we could then save the two images to show to the user at the end (not sure if any CI platforms would easily support that though).

I think what we'll want to cover is primarily regressions, while adding new features to the test suite every time a feature is added. It would be great to be able to test multiple features at the same time but that would grow really quick if we had to test every combination.