Snapshot testing

[adrian-goe]

I am currently trying to write a test for a procedural generated map. In order to test this properly, a result is generated that is too large to write by hand or have it written in the testfile.

So something like Snapshot testing would be a nice feature for this. Example: https://jestjs.io/docs/en/snapshot-testing

[astrale-sharp]

I'm going to try to implement this. comparing images in godot is fairly simple

func capture_screen() -> Image:
    get_viewport().set_clear_mode(Viewport.CLEAR_MODE_ONLY_NEXT_FRAME)
    # Wait until the frame has finished before getting the texture.
    yield(VisualServer, "frame_post_draw")

    # Retrieve the captured image.
    var img = get_viewport().get_texture().get_data()

    # Flip it on the y-axis (because it's flipped).
    img.flip_y()
    return img

func _path2img(path) -> Image:
    var img : Image = Image.new()
    img.load("res://my_png.png")
    return img

func _cmp_img(img1, img2) -> bool:
    return img1.get_data() == img2.get_data()

the question remaining would be : how do we want to use this from Gut? something like

func test_the_screen_remains_the_same():
    setup_screen()
    #  implicitly captures the screen and compares it to "my_screen_capture.png"
    assert_snapshot_equals("res://my_screen_capture.png")

something like that would pretty much work no? maybe to be able to capture the screen we could expose capture_image() so we can access it like this:

var img : Image = _strutils.capture_image()
img.save_png("some_pic.png")

what do you think? (im waiting for an answer before i actually start working on gut for this)

[adrian-goe]

Images would of course also be a possibility. But I had rather thought of data, e.g. JSON or text.

Then it would look like this from Gut:

func test_something():
    var worldSize = 100
    var result = WorldGenerator.generate_world(worldSize)

    assert_snapshot_json(result) # or assert_snapshot(result) for raw text?

In jest, a snapshots folder is created. In this folder, the test data is represented as a folder and when a snapshot is taken, the name of the test function is taken as the file name. If no snapshot is used in the test class, it does not have to be created. res://snapshots/worldgeneartor-test/test_something.json

In the event that I have two snapshots in one function, I would expect that files would be created in this way: res://snapshots/worldgeneartor-test/test_something.json res://snapshots/worldgeneartor-test/test_something_1.json The first call would then be without postfix and the second call would then be with postfix

• If no snapshot exists, it is created and the test is successful.
• If one exists, then read and compare.

Maybe then assert_snapshot_json() needs an internal function that converts an object into a JSON string.

For me personally, I don't see any use case for screenshots, since random functions rarely result in a non-deterministic image. But I would expect an equivalent implementation for images. Maybe with an optional path? assert_snapshot_image_equals(result, "res://my_screen_capture.png")

Depending on how the images are compared, it may also be possible to specify to what extent the images must be similar? assert_snapshot_image_equals(result, 0.8, "res://my_screen_capture.png")

[bitwes]

@astrale-sharp I was thinking images at first too. I see now it's really just the object's definition. @adrian-goe It looks like most of the functionality for snapshot testing revolves around creating/updating the file to be compared and automatically linking that file to a test name.

I think most of the comparing logic is done for exact matches. Comparing dictionaries is easy enough with compare_deep. Turning JSON into dictionaries is what JSON.parse does.

The other feature I see that is important is property matchers. Implementing this is going to be a little trickier but should be feasible using the various TYPE_ constants.

I'll add a Requirements comment that can be updated as the discussion continues.

[bitwes]

Requirements

snapshot_directory setting.

• Settable via the GUI and command line.
• Directory will be created by GUT if it doesn't exist at the start of a run or when the first snapshot is encountered in a test.
• This directory would be managed by GUT and should not require user interaction except to remove obsolete snapshots.

Define/compare a snapshot.

This will accept json/dictionary. It will not accept an arbitrary object since there isn't a good standard method for converting an object to json.

• This will create a snapshot from passed in json/dictionary if it doesn't exist.
• If the snapshot exists it will compare the passed in json/dictionary.
• Naming of the snapshot must be unique to the test it is defined in. Something like the below should work. Where _<X> would be a counter of the number of snapshots checked in a test.

  <snapshot_directory>/<file_name>_<inner_class_name>_<test_name>_<X>.snapshot

Syntax

var obj = Foo.new()
assert_matches_snapshot(obj.make_json())

When assert_matches_snapshot is called it will check to see if the snapshot exists. If it does not then it will create it. If it does exist it will compare the snapshot to the passed in json/dictioanry.

When snapshot is created a warning will be printed in the test output. A test run will also track the number of snapshots it created and print a warning at the end indicating how many were created.

Property Matchers

In order to prevent issues with values that are always different, an optional dictionary can be passed to define values that should be compared by type and not value.

var obj = Foo.new()
obj.bar = randi()%10
assert_matches_snapshot(obj.make_json(), {
  "bar":TYPE_INT
})

Only values in the dictionary structure that should not be compared literally should be specified in the dictionary.

Update snapshots.

A manual approach would be to make the user delete any invalid snapshots. This could be a first step. Using the basic approach from Jest GUT could regenerate all the snapshots with a command. I think this would just run ALL tests but would not compare, only generate any snapshots it encounters (or generate and then compare).

Detect unused snapshots.

This could be done on demand or whenever a full test run is performed. GUT should keep track of all the snapshots it used and compare that to the files in the snapshot directory. GUT could either warn about unused snapshots or just delete them.

I think warning on a full test run (one that does not include any restrictions on file/class/test name) and then creating a command that could be run via the command line or GUI to show, then prompt, then remove any unused snapshots.

[astrale-sharp]

Oh! the name snapshot mislead me to believe we were talking about images!

worth noting : JSON.print(a : Variant) converts a variant to a json representation a.to_json() does something similar

[astrale-sharp]

It still seems feasible : the only thing I wouldnt be sure is how to get the information for this <snapshot_directory>/<file_name>_<inner_class_name>_<test_name>_<X>.snapshot from inside the test_* methods this in particular <file_name>_<inner_class_name>_<test_name>

[bitwes]

All that info should be in the _gut instance available to the test. I can look more into this tomorrow.

[astrale-sharp]

I thought I would find gut._inner_class_name to contain the class_name at test time but it doesnt,

• [x]

• [ ]

[astrale-sharp]

Its really a shame Godot doesnt support another way than class_name to register script as classes, giving everything its type would allow for autocompletion && instant documentation, it would really be great (but we cant pollute user space with lots of class_name).

[astrale-sharp]

can I get some help about where to find <inner_class_name> , I researched a lot through the code but I didnt find it.. @bitwes

[bitwes]

This line sets the local variable inside gut:

var the_script = _test_collector.scripts[indexes_to_run[test_indexes]]

the_script is an instance of test_collector.gd/TestScript. This class has the path to the script and the inner_class_name. This information is not available outside of gut though since this is in a local method variable.

I think the best way to make this info available elsewhere would be to add a snapshot_name property to test_collector.gd/Test. This would get populated by test_collector.gd when it populates the tests array. That way the logic can stay in test_collector.gd. Once this data is in the class then it will be accessible via gut.get_current_test_object().snapshot_name.

[lihop]

With regards to visual snapshot testing, I recently created this port of pixelmatch with the intention of using it for visual regression testing. I'm using it with Gut in this file.