croconut / godot-tester

A Github Action to handle testing Godot applications with GUT
MIT License
36 stars 14 forks source link

Catching failed scripts ignored by GUT can lead to false positives #15

Closed db0 closed 2 years ago

db0 commented 2 years ago

E.g. see this

 script error found at SCRIPT ERROR: Invalid get index 'temp_properties_modifiers' (on base: 'previously freed instance').
failed test count increased: 1

This failed because GUT had torn down the test setup, probably while some yields were still waiting. Then when those yields returned, they caused this SCRIPT ERROR. But this is OK, since we don't care about these timing issues during testing. However the tester still reports this as a failure, even through I've set assert-check: true

link: https://github.com/db0/godot-card-game-framework/runs/4340432389?check_suite_focus=true

db0 commented 2 years ago

Hm, even setting minimum-pass: "0.9" is not enough to make this pass :(

croconut commented 2 years ago

okay i'll add an ignore error option. its likely better to resolve this discrepancy outside of the testing framework though.