search

tarantool / test-run

Tarantool functional testing framework
14 stars 15 forks source link

In case `--force` option is set, continue running tests even if one or more are hung #357

Open Serpentian opened 1 year ago

Serpentian commented 1 year ago

test-run --help says that --force option makes the program to "go on with other tests in case of an individual test failure". Currently, if any test hangs, test-run just exits after sending SIGKILL to workers. From the developer prospective of view hanging and failing are the same: something doesn't work. When setting flag, I want to know how many and which tests doesn't work. We're not supposed to stop after hang.

Expected behavior: continue running.

Totktonada commented 1 year ago

Currently, if any test hangs, test-run just exits after sending SIGKILL to workers.

There is a test timeout (--test-timeout, 110 seconds by default) and it should gracefully handle a hung test. Hitting --no-output-timeout (120 seconds by default), where test-run SIGKILLs workers and exits, shouldn't occur normally.

It is possible to handle the no-output timeout gracefully as well. But we anyway need a reproducer for the situation, when you reach the no-output timeout.