search

MovingBlocks / Terasology

Terasology - open source voxel world
http://terasology.org
Apache License 2.0
3.64k stars 1.32k forks source link

introduce JUnit tag "flaky", to exclude tests from running. #5231

Closed soloturn closed 2 months ago

soloturn commented 2 months ago

ExampleTest.testClientCreation fails without real reason, mark it flaky at the moment. permit this tag in engine-tests/build.gadle.kts to separate test runs for normal integration tests, and ones which are not stable on github, for currently unknown reasons. normally of course it would be better to fix these tests - but accepting pull requests is coupled on tests running through. now two commands instead of one run all integration tests:

gradle integrationTest
gradle integrationTestFlaky

please do not lightheartedly mark tests as flaky.

jdrueckert commented 2 months ago

image

The delay manager test fails so rarely that I wouldn't mark that one flaky, but the ExampleTests fail more frequently, so let's mark those flaky. Amusingly the one already marked flaky didn't fail this time :sweat_smile:

soloturn commented 2 months ago

ok, committed your suggestions and squashed into one. i am understanding correct, we still see the cause of any failed integration test?

jdrueckert commented 2 months ago

i am understanding correct, we still see the cause of any failed integration test?

in the sense that test failures are still reported including stack trace etc.? yes but they shouldn't fail the pipeline anymore.