tempesta-tech / tempesta-test

Test suite for Tempesta FW
11 stars 5 forks source link

Ctrl+C doesn't interrupt tests #101

Open midenok opened 5 years ago

midenok commented 5 years ago
    testMethod()
  File "/root/tempesta-test/mixed_requests/test_mixed.py", line 302, in test_pip
eline
    self.routine(pipeline_lua)
  File "/root/tempesta-test/mixed_requests/test_mixed.py", line 298, in routine
    wrk.stop()
  File "/root/tempesta-test/helpers/stateful.py", line 59, in stop
    self.force_stop()
  File "/root/tempesta-test/helpers/stateful.py", line 43, in force_stop
    stop_proc()
  File "/root/tempesta-test/framework/client.py", line 97, in __on_finish
    self.proc_results = self.resq.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get         
    res = self._recv()
  File "/usr/lib/python2.7/unittest/signals.py", line 36, in __call__
    self.default_handler(signum, frame)
KeyboardInterrupt
^C^C
^C

Consider this: http://zguide.zeromq.org/py:interrupt

vankoven commented 5 years ago

Actually it does. But it doesn't happen immediately, graceful interruption is performed instead: current test is completed up to end and then test framework stops. When the tests are started next time they will be started from the next test. Graceful test shutdown is highly important since it's required to clean up environment on 1-3 virtual machines after each test or next tests run wouldn't be possible.

Immediate interruption takes place if user presses Ctrl-C multiple times. In this case no environment cleanup happen and exception is thrown. That was the situation that you have encountered.

It's well described behaviour of python unittest module. Environment changes several times during some tests and allowing to interrupt such tests at any time just makes tests too sophisticated and hard to debug and mantain.

midenok commented 5 years ago

As above picture shows, multiple ^C was pressed and test didn't exit.

vankoven commented 5 years ago

Ok, that wasn't clear from the original issue description. Since you have real issues with starting TempestaFW itself, may be some behaviour on keyboard interrupt signal differs and the framework get stack an some command. It can get stack on unloading Tempesta if it has issues...

The log will be more helpful, if you can reproduce the isue with -vvvvv arguments (level 5 of verbose messages). In this case we will see what cause the freeze.

midenok commented 5 years ago
Registering tempesta tempesta
Registering backend nginx
Registering backend deproxy
Registering backend deproxy_echo
Registering backend deproxy_ka
Registering tempesta tempesta_fi
    Run command 'sysctl -w net.ipv4.tcp_tw_reuse=1' on host localhost with environment {}
    Run command 'sysctl -w net.ipv4.tcp_fin_timeout=10' on host localhost with environment {}
    Run command 'sysctl -w net.ipv4.tcp_tw_reuse=1' on host localhost with environment {}
    Run command 'sysctl -w net.ipv4.tcp_fin_timeout=10' on host localhost with environment {}
    Run command 'sysctl -w net.ipv4.tcp_tw_reuse=1' on host localhost with environment {}
    Run command 'sysctl -w net.ipv4.tcp_fin_timeout=10' on host localhost with environment {}
    Run command 'sysctl -w net.ipv4.tcp_max_orphans=1000000' on host localhost with environment {}
Disabled test "regression.test_stress_pipeline.PipelineFaultInjection" : Problems with stap, see tempesta-testa#45
Disabled test "sessions.test_sticky_sess_stress.OneClient" : 
Disabled test "tcp_connection.test_connection_close.CloseClientConnectiononInvalidReq" : Invalid request is parsed as two requests. The second response is not supported by current deproxy architecture.
Disabled test "regression.test_srv_failovering.FailoveringTest.test_on_close" : 
Disabled test "regression.test_srv_failovering.FailoveringTest.test_on_shutdown" : 
Disabled test "regression.test_shutdown.ShutdownTest.test_shutdown" : 
Disabled test "regression.test_shutdown.ShutdownTest.test_shutdown_with_traffic" : 
Disabled test "very_many_backends.test_stress_1M.ManyBackends1InGroup" : Read errors
Disabled test "very_many_backends.test_stress_1M.ManyBackends32InGroup" : Read errors
Disabled test "long_body.test_long_response.ResponseTest1M" : 
Disabled test "long_body.test_long_response.ResponseTest1k" : 
Disabled test "mixed_requests.test_mixed.MixedRequests.test_trace" : Too many 502, sometimes no good answers at all
Disabled test "long_body.test_response_wrong_length.ResponseSmallBodyLength" : 
Disabled test "cache.test_purge.TestPurge.test_purge" : 
Disabled test "sched.test_ratio_dynamic.RatioDynamicMax" : Maximum latency delay is unpredictable in heavy concurrent environment
Disabled test "sched.test_ratio_dynamic.RatioPredictMax" : Maximum latency delay is unpredictable in heavy concurrent environment
Disabled test "sched.test_ratio_dynamic_recalc.RatioDynamicMax" : Maximum latency delay is unpredictable in heavy concurrent environment
Disabled test "sched.test_ratio_dynamic_recalc.RatioPredictMax" : Maximum latency delay is unpredictable in heavy concurrent environment
Disabled test "tls.test_tls_stress.StressTls" : Crash happens, see tempesta-tech/tempesta#1146
Disabled test "reconf.test_stress_grace" : See tempesta-tech/tempesta#1043 and tempesta-tech/tempesta#1044
Disabled test "malformed.test_malformed_headers" : See tempesta-tech/tempesta#1050 and tempesta-tech/tempesta#1053

----------------------------------------------------------------------
Running functional tests (resuming from mixed_requests.test_mixed.MixedRequests.test_post_empty)...
----------------------------------------------------------------------

    Run command 'echo "Start test: mixed_requests.test_mixed.MixedRequests.test_post_empty" > /dev/kmsg' on host localhost with environment {}
test_post_empty (mixed_requests.test_mixed.MixedRequests) ...   Init test case...
    Run command 'date +%s.%N' on host localhost with environment {}
Running deproxy
Running deproxy server manager
    Starting Nginx on nginx
    Run command 'netstat --inet -apn' on host localhost with environment {}
    Listen ['tcp', '0', '0', '0.0.0.0:80', '0.0.0.0:*', 'LISTEN', '173/nginx:', 'master', 'p']
    Listen ['tcp', '0', '0', '127.0.0.53:53', '0.0.0.0:*', 'LISTEN', '149/systemd-resolve']
    Checking 127.0.0.1:8000
    Run command 'nginx -c /tmp/nginx/nginx_nginx.cfg' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Run command 'curl http://127.0.0.1:8000/nginx_status' on host localhost with environment {}
    Starting TempestaFW on localhost
    Run command '/root/tempesta/scripts/tempesta.sh --start' on host localhost with environment {'TFW_CFG_PATH': u'/tmp/tempesta/tempesta.conf'}
Running client
    Run command 'grep -c processor /proc/cpuinfo' on host localhost with environment {}
Root privileges are required: need access to module loading on localhost.
    Run command 'wrk -d 10 -t 1 -c 10 --timeout 60 -s /tmp/client/wrk.lua http://127.0.0.1:80/' on host localhost with environment {}
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/root/tempesta-test/framework/client.py", line 9, in _run_client
    res = remote.client.run_cmd(client.cmd, timeout=(client.duration + 5))
  File "/root/tempesta-test/helpers/remote.py", line 84, in run_cmd
    (err_msg, stdout, stderr))
Exception: Remote error: Error running command 'wrk -d 10 -t 1 -c 10 --timeout 60 -s /tmp/client/wrk.lua http://127.0.0.1:80/' on localhost, stdout = , stderr = /tmp/client/wrk.lua: /tmp/client/wrk.lua:1: attempt to index global 'wrk' (a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)

Stopping client
    Run command 'echo "End test:   mixed_requests.test_mixed.MixedRequests.test_post_empty" > /dev/kmsg' on host localhost with environment {}
Traceback (most recent call last):
  File "./run_tests.py", line 303, in <module>
    result = testRunner.run(testsuite)
  File "/usr/lib/python2.7/unittest/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/root/tempesta-test/mixed_requests/test_mixed.py", line 320, in test_post_empty
    self.routine(lua_post_empty)
  File "/root/tempesta-test/mixed_requests/test_mixed.py", line 298, in routine
    wrk.stop()
  File "/root/tempesta-test/helpers/stateful.py", line 59, in stop
    self.force_stop()
  File "/root/tempesta-test/helpers/stateful.py", line 43, in force_stop
    stop_proc()
  File "/root/tempesta-test/framework/client.py", line 97, in __on_finish
    self.proc_results = self.resq.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
  File "/usr/lib/python2.7/unittest/signals.py", line 36, in __call__
    self.default_handler(signum, frame)
KeyboardInterrupt
krizhanovsky commented 5 years ago

Just got hang on deproxy start error:

test_basic (tls.test_tls_handshake.Tls12) ...   Init test case...
Running deproxy
Running deproxy server manager
    Deproxy: Server: Start on 127.0.0.1:8000.
ERROR
    Teardown
Not started
    Deproxy: Server: Stop on 127.0.0.1:8000.
Exception in stopping process: 'NoneType' object has no attribute 'close'
Stopping deproxy

^C
^C^C^C^C^C^C

It seems we hang on proc.join() if deproxy can not start. ERROR message is quite misleading by the way: the TLS test is buggy, but it's quite hard to use deproxy as a separate component due to lack of good error messages.

krizhanovsky commented 5 years ago

Just got another hang

Running deproxy server manager
    Starting Nginx on 0
    Starting TempestaFW on localhost
Running client
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/root/tempesta-test/framework/client.py", line 9, in _run_client
    res = remote.client.run_cmd(client.cmd, timeout=(client.duration + 5))
  File "/root/tempesta-test/helpers/remote.py", line 84, in run_cmd
    (err_msg, stdout, stderr))
Exception: Remote error: Error running command '/root/wrk/wrk -d 10 -t 2 -c 10 --timeout 60 -s /tmp/client/foo.lua https://127.0.0.1:443/' on localhost, stdout = , stderr = unable to connect to 127.0.0.1:443 Connection refused

Stopping client
^C^CTraceback (most recent call last):
  File "./run_tests.py", line 303, in <module>
    result = testRunner.run(testsuite)
  File "/usr/lib/python2.7/unittest/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/root/tempesta-test/tls/test_tls_stress.py", line 94, in test
    wrk.stop()
  File "/root/tempesta-test/helpers/stateful.py", line 59, in stop
    self.force_stop()
  File "/root/tempesta-test/helpers/stateful.py", line 43, in force_stop
    stop_proc()
  File "/root/tempesta-test/framework/client.py", line 97, in __on_finish
    self.proc_results = self.resq.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
  File "/usr/lib/python2.7/unittest/signals.py", line 36, in __call__
    self.default_handler(signum, frame)
KeyboardInterrupt
^C^C

It becomes annoying to debug new tests with the problem, so I raise priority a little bit.