tarantool / test-run

Tarantool functional testing framework
14 stars 15 forks source link

luatest+app: non-zero exit code is not properly handled #393

Open Totktonada opened 1 year ago

Totktonada commented 1 year ago

Just output from the process is not always enough, especially in presence of a buffering problem #119.

The exit code is not reported anyhow for a core = luatest and core = app tests.

Examples:

-- test/app-luatest/foo_test.lua
local t = require('luatest')
local g = t.group()

g.after_all(function()
    os.exit(1)
end)

g.test_foo = function()
    t.assert_equals(1, 1)
end
#!/usr/bin/env tarantool
-- test/app-tap/foo.test.lua
-- NB: Remember chmod a+x.
os.exit(1)

Let's run the luatest based test:

$ ./test/test-run.py app-luatest/foo_test.lua
<...>
[001] app-luatest/foo_test.lua                                        [ fail ]
[001] Test failed! Output from reject file /tmp/t/rejects/app-luatest/foo.reject:
[001] Tarantool version is 3.0.0-entrypoint-272-g2afde5b1d
[001] TAP version 13
[001] 1..1
[001] # Started on Mon May 29 20:50:44 2023
[001] # Starting group: app-luatest.foo
[001] ok     1  app-luatest.foo.test_foo
[Main process] Got failed test; gently terminate all workers...
[001] Worker "001_app-luatest" got failed test; stopping the server...

All the test cases are passed, but the test is failed. No any word regarding the reason of the fail.

Let's try the app test:

$ ./test/test-run.py app-tap/foo.test.lua
<...>
[001] app-tap/foo.test.lua                                            [ fail ]
[001] Test failed! Output from reject file /tmp/t/rejects/app-tap/foo.reject:
[001] 
[001] Last 15 lines of Tarantool Log file [Instance "app_server"][/tmp/t/001_app-tap/foo.test.lua.tarantool.log]:
[Main process] Got failed test; gently terminate all workers...
[001] Worker "001_app-tap" got failed test; stopping the server...

Quite similar. No information, just '[ fail ]'.