tarantool / test-run

Tarantool functional testing framework
14 stars 14 forks source link

luatest: mixing of stdout and stderr has bad consequences #392

Closed Totktonada closed 1 year ago

Totktonada commented 1 year ago

https://github.com/tarantool/test-run/blob/344becd72c697e950471e5821f32c307a155bcdd/lib/luatest_server.py#L61

Quote from a test output:

ok     10   integration.general.tls:false.test_constants
o2023-05-29 14:44:29.789 [2895436] main/103/luatest/etcd-client.client.pool V> etcd transport | req: /kv/put {"key":"a2V5XzAwMDAwMjk=","value":"dmFsdWVfMzA="}
<...>
2023-05-29 14:44:30.466 [2895436] main/103/luatest/etcd-client.client.pool V> etcd transport | req: /lease/grant {"ID":7587870927308614055,"TTL":60}
k     11    integration.general.tls:false.test_range_fetch_by_prefix
ok     12   integration.general.tls:false.test_range_fetch_range

Mixed in the middle of the 'ok' word.

test-run's diagnostics:

[092] etcd-client/etcd_client_general_integration_te>                 
[092] TAP13 parse failed (Descending test id on line: 'ok     12\tintegration.general.tls:false.test_range_fetch_range').
[092] 
[092] No result file (etcd-client/etcd_client_general_integration.result) found.
[092] Run the test with --update-result option to write the new result file.
[092] [ fail ]

https://github.com/tarantool/test-run/blob/344becd72c697e950471e5821f32c307a155bcdd/lib/pytap13.py#L148-L150

I already suggested to don't mix the streams in various discussions. Now I'm insisting on it.

We should find another way to report stderr output in case of a failure. The mixing into stdout usually works due to the skipping of unparseable lines in the TAP13 parser, but it is an unstable solution.