Installed tests (as if from skupper-router RPM) do not run

[jiridanek]

See Dispatch issue https://issues.apache.org/jira/browse/DISPATCH-1449

There is not an obvious way how to run the tests. The usual method through ctest is not available.

Looks like there is no way at all to run unittests, but that does not matter that much, because we want to test the installed skrouterd binary here. (So, the unittest binaries probably should not be installed and packaged into the -tests RPM; which they apparently are not, so we are good there.)

There is helper script run_system_tests.py. But

% ./run_system_tests.py 
zsh: permission denied: ./run_system_tests.py

% python ./run_system_tests.py
Traceback (most recent call last):
  File "/usr/lib/skupper-router/tests/./run_system_tests.py", line 30, in <module>
    import system_test
  File "/usr/lib/skupper-router/tests/system_test.py", line 63, in <module>
    from skupper_router.management.client import Node
ModuleNotFoundError: No module named 'skupper_router'

Run tests the following way (as root)

cd /usr/lib/skupper-router/tests
mkdir failed passed
for i in system_tests_*.py; do timeout 1m python3 -m unittest $i -v &> testout.txt; res=$?; if [[ $res != 0 ]]; then mv testout.txt failed/$i.txt; else mv testout.txt passed/$i.txt; fi;  done

Many tests fail

% ls passed failed
failed:
system_tests_bad_configuration.py.txt  system_tests_http.py.txt                    system_tests_open_properties.py.txt           system_tests_sasl_plain.py.txt             system_tests_topology.py.txt
system_tests_cmdline_parsing.py.txt    system_tests_http1_adaptor.py.txt           system_tests_policy.py.txt                    system_tests_skmanage.py.txt               system_tests_topology_addition.py.txt
system_tests_delivery_abort.py.txt     system_tests_http1_over_tcp.py.txt          system_tests_policy_oversize_compound.py.txt  system_tests_socket_address_family.py.txt  system_tests_topology_disposition.py.txt
system_tests_drain.py.txt              system_tests_log_message_components.py.txt  system_tests_priority.py.txt                  system_tests_ssl.py.txt                    system_tests_two_routers.py.txt
system_tests_edge_router.py.txt        system_tests_management.py.txt              system_tests_router_annotations.py.txt        system_tests_stuck_deliveries.py.txt       system_tests_user_id.py.txt
system_tests_handle_failover.py.txt    system_tests_one_router.py.txt              system_tests_router_mesh.py.txt               system_tests_tcp_adaptor.py.txt            system_tests_user_id_proxy.py.txt

passed:
system_tests_address_watch.py.txt     system_tests_core_endpoint.py.txt         system_tests_dynamic_terminus.py.txt  system_tests_http2.py.txt             system_tests_policy_oversize_basic.py.txt  system_tests_websockets.py.txt
system_tests_autolinks.py.txt         system_tests_default_distribution.py.txt  system_tests_failover_list.py.txt     system_tests_interior_sync_up.py.txt  system_tests_protocol_settings.py.txt
system_tests_connector_status.py.txt  system_tests_delivery_counts.py.txt       system_tests_grpc.py.txt              system_tests_log_level_update.py.txt  system_tests_routing_protocol.py.txt
system_tests_core_client.py.txt       system_tests_distribution.py.txt          system_tests_heartbeats.py.txt        system_tests_multicast.py.txt         system_tests_skstat.py.txt

For various reasons, oftentimes this one (which should not be too hard to fix, either provide import path, or remove usage of compat.BINARY if possible, or reimplement the function for system tests and put it into system_tests.py)

% cat failed/system_tests_delivery_abort.py.txt
system_tests_delivery_abort (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: system_tests_delivery_abort (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: system_tests_delivery_abort
Traceback (most recent call last):
  File "/usr/lib64/python3.10/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/usr/lib/skupper-router/tests/system_tests_delivery_abort.py", line 24, in <module>
    from skupper_router_internal.compat import BINARY
ModuleNotFoundError: No module named 'skupper_router_internal'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

or this one

======================================================================
ERROR: test_verify_reject_id_with_ctrl_char (system_tests_bad_configuration.RouterTestIdFailChar)
This test case sets up a router using a configuration router id
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/skupper-router/tests/system_tests_bad_configuration.py", line 174, in test_verify_reject_id_with_ctrl_char
    self.do_test_id_fail_char(
  File "/usr/lib/skupper-router/tests/system_tests_bad_configuration.py", line 201, in do_test_id_fail_char
    self.run_router_assert_failure(conf_path, expected_log=expected)
  File "/usr/lib/skupper-router/tests/system_tests_bad_configuration.py", line 210, in run_router_assert_failure
    lib_include_path = pathlib.Path(os.environ["QPID_DISPATCH_HOME"], "python")
  File "/usr/lib64/python3.10/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'QPID_DISPATCH_HOME'

or

======================================================================
ERROR: test_01_config_relative_path (system_tests_cmdline_parsing.CommandLineTest)
Starts the router as daemon, enforcing a config file name with
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/skupper-router/tests/system_tests_cmdline_parsing.py", line 91, in test_01_config_relative_path
    self.run_router_as_daemon("test-router", os.getcwd() + '/test.pid')
  File "/usr/lib/skupper-router/tests/system_tests_cmdline_parsing.py", line 66, in run_router_as_daemon
    [os.path.join(os.environ.get('BUILD_DIR'), 'router', 'skrouterd'), '-d',
  File "/usr/lib64/python3.10/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

or

system_tests_drain (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: system_tests_drain (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: system_tests_drain
Traceback (most recent call last):
  File "/usr/lib64/python3.10/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/usr/lib/skupper-router/tests/system_tests_drain.py", line 27, in <module>
    from system_tests_drain_support import DrainMessagesHandler, DrainOneMessageHandler
ModuleNotFoundError: No module named 'system_tests_drain_support'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

system_tests_edge_router (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: system_tests_edge_router (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: system_tests_edge_router
Traceback (most recent call last):
  File "/usr/lib64/python3.10/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/usr/lib/skupper-router/tests/system_tests_edge_router.py", line 38, in <module>
    from test_broker import FakeBroker
ModuleNotFoundError: No module named 'test_broker'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

======================================================================
ERROR: tearDownClass (system_tests_handle_failover.FailoverTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/skupper-router/tests/system_test.py", line 920, in tearDownClass
    cls.tester.teardown()
  File "/usr/lib/skupper-router/tests/system_test.py", line 870, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 1531262 error: exit code 2, expected -1
/usr/bin/python3 /usr/lib/skupper-router/tests/failoverserver.py -a amqp://127.0.0.1:29047
/usr/lib/skupper-router/tests/system_test.dir/system_tests_handle_failover/FailoverTest/test_4_remove_router_B_connect_to_my_server/python3-7.cmd
>>>>
/usr/bin/python3: can't open file '/usr/lib/skupper-router/tests/failoverserver.py': [Errno 2] No such file or directory

Sometimes they timeout, that's why I have the limit in the command. E.g. system_tests_one_router.py is one of the ones that timeout.

[jiridanek]

Issues to address:

• [ ] make run_system_tests.py executable? (it has hashbang)
• [ ] (unrelated issue, to be filled: consider making skstat and skmanage a shell-script launchers for skstat.py and skmanage.py; dealing with python sources that lack .py extension is annoying)
• [ ] resolve missing / unfound dependencies, such as that /usr/lib/skupper-router/tests/failoverserver.py
• [ ] run the tests in the rpm gha job; that will make the job run quite long, but it should ensure this stays fixed
• [ ] somehow set correct environment for tests, so that they can find their python imports
• [ ] do not require running tests as root (tests create temp files to current dir, /usr is writable only by root), I consider this optional, maybe leave it for next issue

[jiridanek]

This could be extended to have the systemtests run the skupper-router container and therefore test the router as it is containerized. This would be awful for debugging issues, but it would more realistically test what gets deployed. This would work only for some tests, me thinks. Container would have to be run with --network=host and the test tmp dir (where configs are) would have to be mounted into it as a volume.

[jiridanek]

Related issue, run these tests in Testing Farm / Packit

• https://github.com/skupperproject/skupper-router/issues/1013