search

tarantool / test-run

Tarantool functional testing framework
14 stars 15 forks source link

test-run is crashed when default tarantool server is crashed #398

Closed ylobankov closed 1 year ago

ylobankov commented 1 year ago

Tarantool with ASAN enabled

Tarantool 3.0.0-entrypoint-469-g852664b4d
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=TRUE
Compiler: Clang-16.0.6
C_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -msse2 -fsanitize=fuzzer-no-link -fsanitize=address -fsanitize-blacklist=/tarantool/asan/asan.supp  -fmacro-prefix-map=/tarantool=. -std=c11 -Wall -Wextra -fsanitize=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -fno-sanitize-recover=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror
CXX_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -msse2 -fsanitize=fuzzer-no-link -fsanitize=address -fsanitize-blacklist=/tarantool/asan/asan.supp  -fmacro-prefix-map=/tarantool=. -std=c++11 -Wall -Wextra -fsanitize=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -fno-sanitize-recover=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror

Steps to reproduce:

  1. Clone tarantool, init and update submodules recursively.
  2. Remove the test-release-asan: CMAKE_ENV = CC=clang-11 CXX=clang++-11 line from the .test.mk file.
  3. Execute the following commands: 
    docker run -it --rm -e HOST_USER_ID=$(id -u) -v $(pwd):/tarantool tarantool/testing:ubuntu-jammy-clang16
useradd -u ${HOST_USER_ID} -m tarantool
cd /tarantool/ && su tarantool
CC=clang-16 CXX=clang++-16 TEST_RUN_EXTRA_PARAMS=app/fiber.test.lua make -f .test.mk test-release-asan

Expected result:

test-run is not crashed, no Python tracebacks, tarantool server's crash is reported.

Actual result:

test-run is crashed with Python tracebacks, tarantool server's crash is not reported.

======================================================================================
WORKR TEST                                            PARAMS          RESULT
---------------------------------------------------------------------------------
[001] Worker "001_app" cannot start tarantool server; the tasks will be ignored...
[001] The raised exception is 'Broken tarantool console handshake' of type '<class 'RuntimeError'>'.
[001] Worker "001_app" received the following error:
[001] Traceback (most recent call last):
[001]   File "/tarantool/test-run/lib/worker.py", line 283, in __init__
[001]     self.inspector = suite.start_server(self.server)
[001]   File "/tarantool/test-run/lib/test_suite.py", line 244, in start_server
[001]     server.deploy(silent=False)
[001]   File "/tarantool/test-run/lib/tarantool_server.py", line 784, in deploy
[001]     self.start(silent=silent, **kwargs)
[001]   File "/tarantool/test-run/lib/tarantool_server.py", line 911, in start
[001]     self.wait_until_started(wait_load, deadline)
[001]   File "/tarantool/test-run/lib/tarantool_server.py", line 1157, in wait_until_started
[001]     ans = yaml.safe_load(temp.execute('box.info.status'))[0]
[001]   File "/tarantool/test-run/lib/tarantool_connection.py", line 161, in execute
[001]     self.opt_reconnect()
[001]   File "/tarantool/test-run/lib/tarantool_connection.py", line 150, in opt_reconnect
[001]     self.reconnect()
[001]   File "/tarantool/test-run/lib/tarantool_connection.py", line 138, in reconnect
[001]     self.connect()
[001]   File "/tarantool/test-run/lib/admin_connection.py", line 96, in connect
[001]     raise RuntimeError('Broken tarantool console handshake')
[001] RuntimeError: Broken tarantool console handshake
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/tarantool/test-run/lib/test.py", line 35, in _run
    self.callable(*self.callable_args, **self.callable_kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 967, in crash_detect
    self.kill_current_test()
  File "/tarantool/test-run/lib/tarantool_server.py", line 1032, in kill_current_test
    if self.current_test.current_test_greenlet:
AttributeError: 'NoneType' object has no attribute 'current_test_greenlet'
2023-07-05T09:10:00Z <TestRunGreenlet at 0x7f733a5e0160 info='Crash detector: <Popen: returncode: None args: ['/tarantool/extra/dist/tarantoolctl', 'start...>'> failed with AttributeError

Process Process-1:
Traceback (most recent call last):
  File "/tarantool/test-run/lib/worker.py", line 283, in __init__
    self.inspector = suite.start_server(self.server)
  File "/tarantool/test-run/lib/test_suite.py", line 244, in start_server
    server.deploy(silent=False)
  File "/tarantool/test-run/lib/tarantool_server.py", line 784, in deploy
    self.start(silent=silent, **kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 911, in start
    self.wait_until_started(wait_load, deadline)
  File "/tarantool/test-run/lib/tarantool_server.py", line 1157, in wait_until_started
    ans = yaml.safe_load(temp.execute('box.info.status'))[0]
  File "/tarantool/test-run/lib/tarantool_connection.py", line 161, in execute
    self.opt_reconnect()
  File "/tarantool/test-run/lib/tarantool_connection.py", line 150, in opt_reconnect
    self.reconnect()
  File "/tarantool/test-run/lib/tarantool_connection.py", line 138, in reconnect
    self.connect()
  File "/tarantool/test-run/lib/admin_connection.py", line 96, in connect
    raise RuntimeError('Broken tarantool console handshake')
RuntimeError: Broken tarantool console handshake

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/tarantool/test-run/dispatcher.py", line 386, in _run_worker
    worker = self.gen_worker(worker_id)
  File "/tarantool/test-run/lib/worker.py", line 297, in __init__
    self.stop_server(cleanup=False)
  File "/tarantool/test-run/lib/worker.py", line 301, in stop_server
    self.suite.stop_server(self.server, self.inspector, silent=silent,
  File "/tarantool/test-run/lib/test_suite.py", line 249, in stop_server
    server.stop(silent=silent)
  File "/tarantool/test-run/lib/tarantool_server.py", line 1092, in stop
    save_join(self.crash_detector)
  File "/tarantool/test-run/lib/tarantool_server.py", line 57, in save_join
    green_obj.get(timeout=timeout)
  File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/home/tarantool/.local/lib/python3.10/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/tarantool/test-run/lib/test.py", line 35, in _run
    self.callable(*self.callable_args, **self.callable_kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 967, in crash_detect
    self.kill_current_test()
  File "/tarantool/test-run/lib/tarantool_server.py", line 1032, in kill_current_test
    if self.current_test.current_test_greenlet:
AttributeError: 'NoneType' object has no attribute 'current_test_greenlet'