tarantool / test-run

Tarantool functional testing framework
14 stars 14 forks source link

Running a bunch of tests with `yes` is broken #437

Closed Serpentian closed 1 month ago

Serpentian commented 1 month ago

Commit https://github.com/tarantool/test-run/commit/3b0ccd0721e5fc772faeab2481f52723d5b5a69d breaks running tests with:

./test-run.py --builddir=.. $(yes 7253 | head -100)

After this commit only one test is launched, which is unacceptable, as It's the only way to check, whether the test is flaky

Totktonada commented 1 month ago

Confirmed. Sorry for the inconvenience.

A temporary workaround is the following (needs patching test-run).

diff --git a/lib/luatest_server.py b/lib/luatest_server.py
index f6520ad..60f29a2 100644
--- a/lib/luatest_server.py
+++ b/lib/luatest_server.py
@@ -180,7 +180,8 @@ class LuatestServer(Server):
         for test_name in glob.glob(os.path.join(suite_path, '*_test.lua')):
             # If neither of the include patterns are substrings of
             # the given test name, skip the test.
-            if not any(p in test_name for p in include_patterns):
+            match_count = [p in test_name for p in include_patterns].count(True)
+            if match_count == 0:
                 continue

             # If at least one of the exclude patterns is a
@@ -209,13 +210,15 @@ class LuatestServer(Server):
                 prefix_len = len(os.path.commonprefix(test_cases))

                 for test_case in test_cases:
-                    tests.append(LuatestTest(test_name, test_suite.args, test_suite.ini,
-                                             params={"test_case": test_case},
-                                             conf_name=test_case[prefix_len:]))
+                    for _ in range(match_count):
+                        tests.append(LuatestTest(test_name, test_suite.args, test_suite.ini,
+                                                 params={"test_case": test_case},
+                                                 conf_name=test_case[prefix_len:]))
             else:
                 # If the test has no 'parallel' tag, run all the
                 # test cases as one task.
-                tests.append(LuatestTest(test_name, test_suite.args, test_suite.ini))
+                for _ in range(match_count):
+                    tests.append(LuatestTest(test_name, test_suite.args, test_suite.ini))

         tests.sort(key=lambda t: t.name)