Xrayez
commented
4 years ago Test contexts
The minimal testing context was introduced in godotengine/godot#40980 without rendering capabilities, but has been working alright for unit testing specifically so far.
The way I see it, it may be feasible to just introduce another integration test context manually. I've previously attempted to create test contexts using doctest's dynamic filtering in https://github.com/Xrayez/godot/tree/test-contexts, but it may be too complex to maintain, and error-prone.
The main challenge is being able to register setup/teardown methods with doctest, which is not a feature of doctest (without code duplication). The suggested setup/teardown mechanism in doctest is to use SUBCASEs, but I think that works better for avoiding duplication in a test case itself, and not really the test environment.
The entry point for unit and integration testing could be rewritten to accept things like:
--test unit--test integration--test project--test rendering
This way, I think it would be still possible to use doctest for those (like godotengine/godot#42938). It means that the entry point would go through additional interface layer, so to speak.
This kind of setup would also help #1533 because it means no compatibility breakage would have to be done in the first place. But godotengine/godot#40148 didn't preserve compatibility with the old tests.
Graphical and UI code testing
I think testing graphical and UI code requires a MainLoop to be running. It's totally possible to feed input events via code, as seen in Xrayez/godot-testbed#5:
extends "res://addons/gut/test.gd"
# https://github.com/godotengine/godot/issues/32597
class TabContainerGuiInputCrash extends TabContainer:
var ev = InputEventMouseButton.new()
func _ready():
var pm := PopupMenu.new()
set_popup(pm)
pm.queue_free()
yield(get_tree(), "idle_frame")
yield(get_tree(), "idle_frame")
yield(get_tree(), "idle_frame")
ev.pressed = true
ev.button_index = BUTTON_LEFT
ev.button_mask = BUTTON_LEFT
ev.position = Vector2(0, 14)
Input.parse_input_event(ev)
yield(get_tree(), "idle_frame")
yield(get_tree(), "idle_frame")
Input.parse_input_event(ev)
Input.parse_input_event(ev)
var container
func setup():
var gut_window = get_parent().get_node('Gut')
gut_window.hide() # need to hide to properly detect input event
container = TabContainerGuiInputCrash.new()
add_child(container)
func test_tab_container_gui_input():
yield(yield_for(1.0, 'Hopefully no crash happens.'), YIELD)
assert_true(true, "No crash, great!")
func teardown():
container.queue_free()The --fixed-fps and --disable-render-loop command-line options could potentially be used to speed up simulation and controlling the rendering loop via code with RenderingServer.force_draw(). See also godotengine/godot#43260, I'm not sure whether those methods would be actually useful for this.
3. Screenshots are compared (by the engine or external script) to the reference images, and marked for manual inspection if they substantial differences (by adding a thick, red border to the image for example).
doctest could be used for this as for GDScript integration tests #1429, but may be overkill, so perhaps an extra step would be indeed required to do this.
But in theory, all this could be done from within a Godot project running on CI. This is where testing frameworks like GUT shine, in my opinion. For instance, I've been successfully running unit tests in Goost, but we still need a way to render stuff on CI.
c0d1f1ed
bruvzg
fire
Calinou
nikitalita
mariomadproductions
Describe the project you are working on: Godot engine.
Describe the problem or limitation you are having in your project: Unit testing was introduced in godotengine/godot#40148, but currently there's no possibility to automatically test any GUI and rendering related code.
Related proposal: #1307 (testing contexts), #1533 (old tests had at least some rendering and UI tests)
Describe the feature / enhancement and how it helps to overcome the problem or limitation: Implement off-screen
DisplayServerfor use on headless CI, and make it compatible with software Vulkan (SwiftShader) / OpenGL (OSMesa) implementations to run on CI without GPU, and add testing framework context with active rendering pipeline (initialized display and rendering servers, and normal project main loop).Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
If this enhancement will not be used often, can it be worked around with a few lines of script?: It can be used as part CI to detect rendering, physics and GUI regressions, and can be used to quickly test specific hardware or driver versions for rendering issues (the same context should be usable with normal
DisplayServers as well).Is there a reason why this should be core and not an add-on in the asset library?: It should be possible to achieve this with module or GDScript project, but probably better to have testing related stuff in the core for cleaner CI configs and to avoid duplicate code in multiple test projects.