godotengine / godot-tests

Repository for Godot benchmarks, regression tests, etc.
MIT License
34 stars 33 forks source link

sample qa tests, repository layout #3

Closed brainsick closed 6 years ago

brainsick commented 6 years ago

Inspired by #2641. General quality assurance discussion here.

Issues:

To Do:

Classes String Test

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests$ time ../brainsick-godot/bin/godot.x11.tools.64 -s tests/qa/classes/StringTest.gd
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
SCRIPT ERROR: test_get_basename_unix_file_sans_extension: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:123.
SCRIPT ERROR: test_get_basename_unix_dir: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:129.
SCRIPT ERROR: test_get_basename_windows_file_sans_extension: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:135.
SCRIPT ERROR: test_get_basename_windows_dir: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:141.
SCRIPT ERROR: test_get_extension_unix_root: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:151.
SCRIPT ERROR: test_get_extension_unix_file_sans_extension: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:154.
SCRIPT ERROR: test_get_extension_unix_dir: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:160.
SCRIPT ERROR: test_get_extension_windows_root: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:163.
SCRIPT ERROR: test_get_extension_windows_file_sans_extension: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:166.
SCRIPT ERROR: test_get_extension_windows_dir: Assertion failed.
   At: res://tests/qa/classes/StringTest.gd:172.

real    0m1.153s
user    0m0.157s
sys 0m0.029s

This spawned issue godotengine/godot#15471.

Classes UndoRedo Test

Inspired by godotengine/godot#11785.

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests$ time ../brainsick-godot/bin/godot.x11.tools.64 -s tests/qa/classes/UndoRedoTest.gd 
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
SCRIPT ERROR: test_create_create_bug: Assertion failed.
   At: res://tests/qa/classes/UndoRedoTest.gd:77.
ERROR: clear_history: Condition ' action_level > 0 ' is true.
   At: core/undo_redo.cpp:326.

real    0m1.161s
user    0m0.163s
sys 0m0.025s

This spawned issue godotengine/godot#15424. This spawned issue godotengine/godot#15460. This spawned issue godotengine/godot#15506.

Issue godotengine/godot#14799 Test

Just an example. This probably belongs in a Class Test.

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests/tests/qa$ time ~/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64 -s issues/14799.gd 
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
SCRIPT ERROR: test_14799: Assertion failed.
   At: res://issues/14799.gd:10.

real    0m1.118s
user    0m0.078s
sys 0m0.020s

Issue godotengine/godot#15166 Test - Shader Crash

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests/tests/qa$ time ~/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64 -s issues/15166.gd 
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
ERROR: operator[]: FATAL: Index p_index=1 out of size (size()=1)
   At: core/vector.h:146.
handle_crash: Program crashed with signal 4
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x37140) [0x7f55fa419140] (??:0)
[2] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0x482e7b] (??:0)
[3] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0xa224c5] (??:?)
[4] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0xa22a0c] (??:?)
[5] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0x9cfee3] (??:?)
[6] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0x18933b7] (??:?)
[7] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0x4c04f5] (??:0)
[8] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0x44f38d] (??:0)
[9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f55fa4031c1] (??:0)
[10] /home/todd/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64() [0x45ad3f] (??:0)
-- END OF BACKTRACE --
Aborted (core dumped)

real    0m0.413s
user    0m0.207s
sys 0m0.042s

Issue godotengine/godot#15149 Test - Wireframe Mode

The beginning of a simple image comparison test. Could be combined with the shader test to monitor shader output.

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests$ time ../brainsick-godot/bin/godot.x11.tools.64 -s tests/qa/issues/15149.gd
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile

real    0m1.151s
user    0m0.709s
sys 0m0.048s

15149 1_before_image 15149 2_after_image

This spawned issue godotengine/godot#15378.

Issue godotengine/godot#15378 Test - PoolDataArray comparisons

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests/tests/qa$ time ~/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64 -s issues/15378.gd 
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
SCRIPT ERROR: test_array_comparison: Assertion failed.
   At: res://issues/15378.gd:48.
SCRIPT ERROR: test_pba_comparison: Assertion failed.
   At: res://issues/15378.gd:58.
SCRIPT ERROR: test_image_pba_comparison: Assertion failed.
   At: res://issues/15378.gd:68.

real    0m1.112s
user    0m0.091s
sys 0m0.013s

This spawned issue godotengine/godot#15374.

Issue godotengine/godot#15374 Test - Image docs and resize crash

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests$ time ../brainsick-godot/bin/godot.x11.tools.64 -s tests/qa/issues/15374.gd
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x37140) [0x7f2ef3fed140] (??:0)
[2] ../brainsick-godot/bin/godot.x11.tools.64(+0x249076d) [0x55920e59876d] (??:0)
[3] Image::resize(int, int, Image::Interpolation) (??:0)
[4] MethodBind3<int, int, Image::Interpolation>::call(Object*, Variant const**, int, Variant::CallError&) (??:0)
[5] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[6] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (??:0)
[7] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (??:0)
[8] GDScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[9] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[10] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (??:0)
[11] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (??:0)
[12] GDScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[13] ScriptInstance::call(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (??:0)
[14] MainLoop::iteration(float) (??:0)
[15] Main::iteration() (??:0)
[16] OS_X11::run() (??:0)
[17] ../brainsick-godot/bin/godot.x11.tools.64(main+0xd5) [0x55920ccf67ef] (??:0)
[18] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f2ef3fd71c1] (??:0)
[19] ../brainsick-godot/bin/godot.x11.tools.64(_start+0x2a) [0x55920ccf663a] (??:0)
-- END OF BACKTRACE --
Aborted (core dumped)

real    0m0.678s
user    0m0.293s
sys 0m0.243s

Issue godotengine/godot#15424 Test - UndoRedo crashes

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests$ time ../brainsick-godot/bin/godot.x11.tools.64 -s tests/qa/issues/15424.gd 
No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x37140) [0x7fde5e01e140] (??:0)
[2] Object::get_instance_id() const (??:0)
[3] UndoRedo::add_do_property(Object*, String const&, Variant const&) (??:0)
[4] MethodBind3<Object*, String const&, Variant const&>::call(Object*, Variant const**, int, Variant::CallError&) (??:0)
[5] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[6] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (??:0)
[7] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (??:0)
[8] GDScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[9] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[10] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (??:0)
[11] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (??:0)
[12] GDScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (??:0)
[13] ScriptInstance::call(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (??:0)
[14] MainLoop::iteration(float) (??:0)
[15] Main::iteration() (??:0)
[16] OS_X11::run() (??:0)
[17] ../brainsick-godot/bin/godot.x11.tools.64(main+0xd5) [0x5644a90587ef] (??:0)
[18] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fde5e0081c1] (??:0)
[19] ../brainsick-godot/bin/godot.x11.tools.64(_start+0x2a) [0x5644a905863a] (??:0)
-- END OF BACKTRACE --
Aborted (core dumped)

real    0m0.669s
user    0m0.318s
sys 0m0.205s
brainsick commented 6 years ago

New issue. I'd like the assert() errors to be more descriptive. Better error descriptions will result in better bug reports.

For example:

todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests/tests/qa$ cat utils/Assert.gd
static func assertEquals(a, b):
    if (a != b):
        print(a, ' != ', b)
        assert(a == b) # not the stack info we need
todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests/tests/qa$ cat utils/AssertTest.gd 
extends MainLoop

const Assert = preload('Assert.gd')

# main test body
func _iteration(delta):
    Assert.assertEquals(true, true)
    Assert.assertEquals(true, false)

    Assert.assertEquals('a', 'a')
    Assert.assertEquals('a', 'b')

    Assert.assertEquals(1, 1)
    Assert.assertEquals(1, 2)

    return true # exit MainLoop
todd@todd-ab28d3:~/repos/remote/brainsick-godot-tests/tests/qa$ ~/Downloads/Godot3.0Beta2/Godot_v3.0-beta2_x11.64 -s utils/AssertTest.gd No touch devices found
OpenGL ES 3.0 Renderer: AMD Radeon (TM) R9 Fury Series (AMD FIJI / DRM 3.18.0 / 4.13.0-21-generic, LLVM 5.0.0)
GLES3: max ubo light: 409
GLES3: max ubo reflections: 455, ubo size: 144
ARVR: Registered interface: Native mobile
True != False
SCRIPT ERROR: assertEquals: Assertion failed.
   At: res://utils/Assert.gd:4.
a != b
SCRIPT ERROR: assertEquals: Assertion failed.
   At: res://utils/Assert.gd:4.
1 != 2
SCRIPT ERROR: assertEquals: Assertion failed.
   At: res://utils/Assert.gd:4.

With this we gain the values that failed but we lose the line it happened on and the function it happened in. Doesn't seem like a good idea.

How can we get the values, the offending function, and the offending file/line? I don't see any means of accessing the call stack from GDScript.

brainsick commented 6 years ago

I don't expect to pursue this any further at this time. Closing PR.

nightblade9 commented 6 years ago

Were these unit/integration tests abandoned?

Xrayez commented 3 years ago

Were these unit/integration tests abandoned?

I suppose the continuation of this could be godotengine/godot#40795, I hope that I won't have to close the PR similarly. 😮

brainsick commented 3 years ago

I was very excited to contribute to Godot at the time. I was more effective at finding bugs than fixing them though. This topic got a few people talking and engaged but got nothing from the development team. I moved on.

Calinou commented 3 years ago

doctest was merged in the master branch and we now welcome pull requests adding tests in the main repsoitory. See https://github.com/godotengine/godot/issues/43440 if you are interested in contributing :slightly_smiling_face: