tarantool / test-run

Tarantool functional testing framework
14 stars 15 forks source link

it is not possible to propagate test status skip to test-run #381

Closed ligurio closed 1 year ago

ligurio commented 1 year ago

test-run can run luatest and tap tests, both test of these types could report status in TAP format ^1. TAP specification supports test status 'skip', but test-run doesn't get it, it interprets it as 'passed':

[1] ~/sources/MRG/tarantool$ cat  test/box-tap/sample.test.lua 
#!/usr/bin/env tarantool

print("TAP version 13")
print("1..3")
print("ok 1 - Input file opened")
print("ok 2 - First line of the input valid")
print("ok 3 # SKIP Read the rest of the file")

[1] ~/sources/MRG/tarantool$ ./test/test-run.py box-tap/sample.test.lua --verbose                                                           

<snipped>

Detected build mode: Release

======================================================================================
WORKR TEST                                            PARAMS          RESULT
---------------------------------------------------------------------------------
[001] box-tap/sample.test.lua                                         
[001] TAP version 13
[001] 1..3
[001] ok 1 - Input file opened
[001] ok 2 - First line of the input valid
[001] ok 3 # SKIP Read the rest of the file
[001] [ pass ]
---------------------------------------------------------------------------------
Top 10 tests by occupied memory (RSS, Mb):

P.S.: This status could be useful in test wrappers that I made in tarantool-ee for etcd-client tests, see https://github.com/tarantool/tarantool-ee/pull/444.

Totktonada commented 1 year ago

[001] TAP version 13 [001] 1..3 [001] ok 1 - Input file opened [001] ok 2 - First line of the input valid [001] ok 3 # SKIP Read the rest of the file [001] [ pass ]

The [ pass ] status is expected here: there are two passed and one skipped test cases.

Did you meant an example like the following?

[001] TAP version 13 [001] 1..3 [001] ok 1 # SKIP [001] ok 2 # SKIP [001] ok 3 # SKIP [001] [ pass ]

Here it is possibly should be [ skip ].