search

tarantool / luatest

Tarantool test framework written in Lua
https://www.tarantool.io/en/doc/latest/reference/reference_rock/luatest/luatest_overview/
Other
40 stars 11 forks source link

File .result is either absent or empty #193

Open Gerold103 opened 2 years ago

Gerold103 commented 2 years ago

If a test formally passes, it does not mean it passes factually. It might be passing while it should not. Then I want to add some debug prints. And I won't see them because .result file is not left after a successful run.

Another problem is that the file is just empty if the test hangs. Unless I do crutches like printing a too big buffer to cause output flush (but I didn't try that, was only said it could work).

Totktonada commented 2 years ago

If a test formally passes, it does not mean it passes factually. It might be passing while it should not.

Is it a problem in luatest or description of problem in a test?

Or you just want to highlight that it is hard to get result of a test that is run in luatest under test-run?

Pure lutest -c -v -p <part_of_test_name> works for you?

Gerold103 commented 2 years ago

Is it a problem in luatest

It is a problem of luatest. In diff tests the .result file is populated while the test runs, at realtime. And it is kept in the end after the test is done. In a form of .reject file at least.

Or you just want to highlight that it is hard to get result of a test that is run in luatest under test-run?

I use it with test-run because it is easier than specifying command line options and environment variables. Hence I want to highlight test-run.

Gerold103 commented 2 years ago

This behaviour also leads to false-positive warnings when the test runs too long. I have a test which runs 30 test cases, each runs from 0.2 to 0.4s. But in the test-run output I see No output during 10 seconds. Will abort after 120 seconds without output. There could be output saying which of the test cases already passed, the tests do not actually hang for 10 seconds without any activity.