Create a screenshot based set of tests for jMonkeyEngine

richardTingle commented 5 months ago

As discussed at https://hub.jmonkeyengine.org/t/screenshot-based-jme-testing/47685/18 I will be creating a set of automated tests that can run as part of the build on standardised hardware. The purpose of this is to reduce the amount of human testing required to assure ourselves that a new JME version has not broken some graphics based behaviour (which is difficult to test under normal automated testing)

Objectives:

Human readable html report including before and after screenshots for manual review
Diff screenshot showing where in the image there is a change
Easy (documented) mechanism for accepting changes
Failed pipeline on test fail
Ability for some tests to be marked as "non deterministic" and only triggering manual human review, not a failed test
Ability for some tests to be marked as "known broken" and only triggering manual human review, not a failed test

richardTingle commented 5 months ago

(I don't think I have rights to assign myself to this issue, but for the avoidance of doubt I'm working on this)

MeFisto94 commented 5 months ago

It seems you already worked on that, but just a heads up that I did so, too, a few years ago. It could even run in docker using some CPU pipeline: https://github.com/MeFisto94/jme3-testing

richardTingle commented 5 months ago

@MeFisto94 Thank you for sharing that, thats interesting. Anything you think I need to watch out for from your previous experience? Did your docker version use cpu openGL rendering as well?

jMonkeyEngine / jmonkeyengine

Create a screenshot based set of tests for jMonkeyEngine #2279