search

tarantool / test-run

Tarantool functional testing framework
14 stars 17 forks source link

Something is broken in test auto-rerun - sometimes it is infinite #335

Closed Gerold103 closed 2 years ago

Gerold103 commented 2 years ago

Getting this output:

[001] [Instance "master"] Failed to start tarantool instance "election_replica3"
[001] [ fail ]
[001] Test "replication/gh-5445-leader-inconsistency.test.lua", conf: "None"
[001]   from "fragile" list failed, rerunning ...
[001] replication/gh-5445-leader-inconsistency.test.>                 
[001] [Instance "master"] Failed to start tarantool instance "election_replica2"
[001] [ fail ]
[001] Test "replication/gh-5445-leader-inconsistency.test.lua", conf: "None"
[001]   from "fragile" list failed, rerunning ...
[001] replication/gh-5445-leader-inconsistency.test.>                 
[001] [Instance "master"] Failed to start tarantool instance "election_replica3"
[001] [ fail ]
[001] Test "replication/gh-5445-leader-inconsistency.test.lua", conf: "None"
[001]   from "fragile" list failed, rerunning ...
[001] replication/gh-5445-leader-inconsistency.test.>                 
[001] [Instance "master"] Failed to start tarantool instance "election_replica2"
[001] [ fail ]
[001] Test "replication/gh-5445-leader-inconsistency.test.lua", conf: "None"
[001]   from "fragile" list failed, rerunning ...
[001] replication/gh-5445-leader-inconsistency.test.>                 
[001] [Instance "master"] Failed to start tarantool instance "election_replica3"
[001] [ fail ]
[001] Test "replication/gh-5445-leader-inconsistency.test.lua", conf: "None"
[001]   from "fragile" list failed, rerunning ...
[001] replication/gh-5445-leader-inconsistency.test.>                 
[001] [Instance "master"] Failed to start tarantool instance "election_replica2"
[001] [ fail ]

on a test. There is a crash happening and I would expect at most 3 retries, like specified in test-run source code. But it seems to be doing them infinitely.

ylobankov commented 2 years ago

This test is from the fragile list, so it will be rerun this amount of times. If you look more carefully, you will see this:

[001] Test "replication/gh-5445-leader-inconsistency.test.lua", conf: "None"
[001]   from "fragile" list failed, rerunning ...

Tests that are marked as not 'fragile' will be rerun only 3 times. In the future we will add an option to tune this. If you want to tune the behaviour for fragile tests, please update the suite.ini file.