godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
91.3k stars 21.24k forks source link

GDScript: Support tracking multiple analyzer and runtime errors in tests #99490

Open dalexeev opened 1 day ago

dalexeev commented 1 day ago

Currently, the GDScript test system has a problem: it does not allow us to track multiple errors. Because of this, we have to have many small files for error tests, unlike warning and feature tests.

While this limitation is justified for parser tests due to cascading errors, I think there is no obstacle for analyzer and runtime tests to track multiple errors. Probably, using = instead of += in modules/gdscript/tests/gdscript_test_runner.cpp is even a bug, leading to overwriting the output, so we only track the last error.

This PR allows:

  1. Track analyzer warnings that occur before runtime errors.
  2. Track core runtime errors that occur before GDScript runtime errors.
  3. Track multiple GDScript analyzer errors.
  4. Track multiple GDScript runtime errors. This is possible because without an active debugger, only the current function is terminated on a runtime error, and the caller continues execution.

How to make multiple subtests of runtime errors within one test:

func subtest_1():
    # Code with one runtime error...

func subtest_2():
    # Code with one runtime error...

func subtest_3():
    # Code with one runtime error...

func test():
    subtest_1()
    subtest_2()
    subtest_3()

Note: I did not combine all groups of similar tests within this PR.

Repiteo commented 16 hours ago

So warnings now have use a ~~ prefix instead of >>? Is there a functional benefit to that, or is it wholly stylistic?

dalexeev commented 16 hours ago

So warnings now have use a ~~ prefix instead of >>? Is there a functional benefit to that, or is it wholly stylistic?

Yes, warnings are not produced in release builds, so we need a marker to cut only them but leave errors.

https://github.com/godotengine/godot/blob/f86dcd4e67a7e715233cc87a265e8ef79a27348d/modules/gdscript/tests/gdscript_test_runner.cpp#L173-L181

This doesn't matter now, since GDScript tests don't seem to run for release builds.