tarantool / test-run

Tarantool functional testing framework
14 stars 15 forks source link

Introduce flaky tests statistics #374

Closed ylobankov closed 1 year ago

ylobankov commented 1 year ago

Example:

$ ./test-run.py --force replication-luatest/
Started ./test-run.py --force replication-luatest/
Running in parallel with 16 workers

... <other tests>

[016] replication-luatest/gh_6033_box_promote_demote>                 [ pass ]
[014] replication-luatest/gh_5418_qsync_with_anon_te>                 [ pass ]
[012] replication-luatest/quorum_misc_test.lua                        [ pass ]
[005] replication-luatest/linearizable_test.lua                       [ fail ]

... <traceback>

[005] Test "replication-luatest/linearizable_test.lua", conf: "None"
[005]   failed, rerunning ...
[005] replication-luatest/linearizable_test.lua                       [ pass ]

---------------------------------------------------------------------------------
Top 10 longest tests (seconds):
*  67.86 replication-luatest/quorum_orphan_test.lua
*  26.00 replication-luatest/quorum_misc_test.lua
*  15.59 replication-luatest/gh_5418_qsync_with_anon_test.lua
*  13.39 replication-luatest/election_fencing_test.lua
*   9.77 replication-luatest/gh_5295_split_brain_test.lua
*   6.95 replication-luatest/no_quorum_test.lua
*   6.29 replication-luatest/bootstrap_strategy_auto_test.lua
*   6.26 replication-luatest/gh_5568_read_only_reason_test.lua
*   5.31 replication-luatest/gh_7253_election_long_wal_write_test.lua
*   5.13 replication-luatest/gh_7086_box_issue_promote_assert_test.lua
---------------------------------------------------------------------------------
Statistics:
* pass: 37 (flaky: 1)
Flaked tasks:
- [replication-luatest/linearizable_test.lua, null]
# logfile:        /private/tmp/t/log/005_replication-luatest.log
# reproduce file: /private/tmp/t/reproduce/005_replication-luatest.list.yaml

$ ls /tmp/t/artifacts/
005_replication-luatest/ log/                     reproduce/               statistics/

Closes #358 Part of tarantool/tarantool-qa#299