owncloud / client

🖥️ Desktop Syncing Client for ownCloud
GNU General Public License v2.0
1.4k stars 664 forks source link

[gui-tests][full-ci] drop test retry #11953

Closed saw-jan closed 1 month ago

saw-jan commented 1 month ago

Feature to retry failed scenario was added in https://github.com/owncloud/client/pull/11920 but unfortunately with retry enabled, if there're setup level fatal errors while running the tests, we might not notice that in CI because with retry enabled even with fatal error the exit code will be 0 that lets the CI to pass.

The weight of not noticing the CI about such failure (which leads to false positives) is heavier than a retry feature. So let's drop the retry feature.

Screenshot from 2024-10-23 13-34-17

https://github.com/owncloud/client/pull/11941#issuecomment-2428396827

I found out that when we run squish test with --retry 1 and if there's some fatal error during loading some modules then the exit code will become 0 instead of 1 or other Same fata error fails the ci if not run with --retry 1: https://drone.owncloud.com/owncloud/client/19730

I have raised this topic in squish forum: https://forum.qt.io/topic/159369/squish-python-squishrunner-returns-zero-0-exit-code-when-using-retry-1-even-if-there-are-fatal-errors