tarantool / test-run

Tarantool functional testing framework
14 stars 15 forks source link

Fix test-run crash when default server is crashed #405

Closed ylobankov closed 1 year ago

ylobankov commented 1 year ago

Before this change, test-run crashed if the default tarantool server had crashed with a segmentation fault, and as a result, this caused an unhandled error with the broken tarantool console handshake. So such a situation wasn't properly processed by test-run and it failed with the following traceback:

Traceback (most recent call last):
  File "/tarantool/test-run/lib/worker.py", line 283, in __init__
    self.inspector = suite.start_server(self.server)
  File "/tarantool/test-run/lib/test_suite.py", line 244, in start_server
    server.deploy(silent=False)
  File "/tarantool/test-run/lib/tarantool_server.py", line 784, in deploy
    self.start(silent=silent, **kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 911, in start
    self.wait_until_started(wait_load, deadline)
  File "/tarantool/test-run/lib/tarantool_server.py", line 1157, in wait_until_started
    ans = yaml.safe_load(temp.execute('box.info.status'))[0]
  File "/tarantool/test-run/lib/tarantool_connection.py", line 161, in execute
    self.opt_reconnect()
  File "/tarantool/test-run/lib/tarantool_connection.py", line 150, in opt_reconnect
    self.reconnect()
  File "/tarantool/test-run/lib/tarantool_connection.py", line 138, in reconnect
    self.connect()
  File "/tarantool/test-run/lib/admin_connection.py", line 96, in connect
    raise RuntimeError('Broken tarantool console handshake')
RuntimeError: Broken tarantool console handshake

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/tarantool/test-run/dispatcher.py", line 386, in _run_worker
    worker = self.gen_worker(worker_id)
  File "/tarantool/test-run/lib/worker.py", line 297, in __init__
    self.stop_server(cleanup=False)
  File "/tarantool/test-run/lib/worker.py", line 301, in stop_server
    self.suite.stop_server(self.server, self.inspector, silent=silent,
  File "/tarantool/test-run/lib/test_suite.py", line 249, in stop_server
    server.stop(silent=silent)
  File "/tarantool/test-run/lib/tarantool_server.py", line 1092, in stop
    save_join(self.crash_detector)
  File "/tarantool/test-run/lib/tarantool_server.py", line 57, in save_join
    green_obj.get(timeout=timeout)
  File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/home/tarantool/.local/lib/python3.10/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/tarantool/test-run/lib/test.py", line 35, in _run
    self.callable(*self.callable_args, **self.callable_kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 967, in crash_detect
    self.kill_current_test()
  File "/tarantool/test-run/lib/tarantool_server.py", line 1032, in kill_current_test
    if self.current_test.current_test_greenlet:
AttributeError: 'NoneType' object has no attribute 'current_test_greenlet'

Now test-run handles the segmentation fault of the default server and the error with the broken tarantool console handshake properly.

Fixes #398