search

dd010101 / vyos-jenkins

Scripts for building custom VyOS stream (1.5 circinus) packages/images. Also legacy scripts for building frozen 1.3 equuleus/1.4 sagitta packages/images.
94 stars 30 forks source link

ISO testd fails test_accel #33

Closed koljenovic closed 1 month ago

koljenovic commented 4 months ago

DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_vpn_pptp.py DEBUG - test_accel_ipv4_pool (main.TestVPNPPTPServer.test_accel_ipv4_pool) ... ok DEBUG - test_accel_ipv6_pool (main.TestVPNPPTPServer.test_accel_ipv6_pool) ... ok DEBUG - test_accel_limits (main.TestVPNPPTPServer.test_accel_limits) ... ok DEBUG - test_accel_local_authentication (main.TestVPNPPTPServer.test_accel_local_authentication) ... ok DEBUG - test_accel_log_level (main.TestVPNPPTPServer.test_accel_log_level) ... FAIL DEBUG - test_accel_name_servers (main.TestVPNPPTPServer.test_accel_name_servers) ... FAIL DEBUG - test_accel_next_pool (main.TestVPNPPTPServer.test_accel_next_pool) ... FAIL DEBUG - test_accel_ppp_options (main.TestVPNPPTPServer.test_accel_ppp_options) ... ok DEBUG - test_accel_radius_authentication (main.TestVPNPPTPServer.test_accel_radius_authentication) ... ok DEBUG - test_accel_shaper (main.TestVPNPPTPServer.test_accel_shaper) ... ok DEBUG - test_accel_snmp (main.TestVPNPPTPServer.test_accel_snmp) ... ok DEBUG - test_accel_wins_server (main.TestVPNPPTPServer.test_accel_wins_server) ... ok DEBUG - DEBUG - ====================================================================== DEBUG - FAIL: test_accel_log_level (main.TestVPNPPTPServer.test_accel_log_level) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py", line 46, in tearDown DEBUG - self.assertTrue(process_named_running(self._process_name)) DEBUG - AssertionError: None is not true DEBUG - DEBUG - ====================================================================== DEBUG - FAIL: test_accel_name_servers (main.TestVPNPPTPServer.test_accel_name_servers) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py", line 46, in tearDown DEBUG - self.assertTrue(process_named_running(self._process_name)) DEBUG - AssertionError: None is not true DEBUG - DEBUG - ====================================================================== DEBUG - FAIL: test_accel_next_pool (main.TestVPNPPTPServer.test_accel_next_pool) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py", line 46, in tearDown DEBUG - self.assertTrue(process_named_running(self._process_name)) DEBUG - AssertionError: None is not true DEBUG - DEBUG - ---------------------------------------------------------------------- DEBUG - Ran 12 tests in 36.274s DEBUG - DEBUG - FAILED (failures=3)

DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_vpn_l2tp.py DEBUG - test_accel_ipv4_pool (main.TestVPNL2TPServer.test_accel_ipv4_pool) ... ok DEBUG - test_accel_ipv6_pool (main.TestVPNL2TPServer.test_accel_ipv6_pool) ... ok DEBUG - test_accel_limits (main.TestVPNL2TPServer.test_accel_limits) ... ok DEBUG - test_accel_local_authentication (main.TestVPNL2TPServer.test_accel_local_authentication) ... ok DEBUG - test_accel_log_level (main.TestVPNL2TPServer.test_accel_log_level) ... FAIL DEBUG - test_accel_name_servers (main.TestVPNL2TPServer.test_accel_name_servers) ... FAIL DEBUG - test_accel_next_pool (main.TestVPNL2TPServer.test_accel_next_pool) ... FAIL DEBUG - test_accel_ppp_options (main.TestVPNL2TPServer.test_accel_ppp_options) ... ok DEBUG - test_accel_radius_authentication (main.TestVPNL2TPServer.test_accel_radius_authentication) ... ok DEBUG - test_accel_shaper (main.TestVPNL2TPServer.test_accel_shaper) ... ok DEBUG - test_accel_snmp (main.TestVPNL2TPServer.test_accel_snmp) ... ok DEBUG - test_accel_wins_server (main.TestVPNL2TPServer.test_accel_wins_server) ... ok DEBUG - test_l2tp_radius_server (main.TestVPNL2TPServer.test_l2tp_radius_server) ... ok DEBUG - test_l2tp_server_authentication_protocols (main.TestVPNL2TPServer.test_l2tp_server_authentication_protocols) ... ok DEBUG - test_vpn_l2tp_dependence_ipsec_swanctl (main.TestVPNL2TPServer.test_vpn_l2tp_dependence_ipsec_swanctl) ... ok DEBUG - DEBUG - ====================================================================== DEBUG - FAIL: test_accel_log_level (main.TestVPNL2TPServer.test_accel_log_level) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py", line 46, in tearDown DEBUG - self.assertTrue(process_named_running(self._process_name)) DEBUG - AssertionError: None is not true DEBUG - DEBUG - ====================================================================== DEBUG - FAIL: test_accel_name_servers (main.TestVPNL2TPServer.test_accel_name_servers) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py", line 46, in tearDown DEBUG - self.assertTrue(process_named_running(self._process_name)) DEBUG - AssertionError: None is not true DEBUG - DEBUG - ====================================================================== DEBUG - FAIL: test_accel_next_pool (main.TestVPNL2TPServer.test_accel_next_pool) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py", line 46, in tearDown DEBUG - self.assertTrue(process_named_running(self._process_name)) DEBUG - AssertionError: None is not true DEBUG - DEBUG - ---------------------------------------------------------------------- DEBUG - Ran 15 tests in 46.821s DEBUG - DEBUG - FAILED (failures=3)

DEBUG - ERROR: One or more tests failed! DEBUG - vyos@vyos:~$ echo EXITCODE:$? DEBUG - echo EXITCODE:$? ERROR - Smoketest failed :/ ERROR - Unknown error occured while VyOS! Traceback (most recent call last): File "/root/vyos-build/scripts/check-qemu-install", line 528, in raise Exception("Smoketest-failed, please look into debug output") Exception: Smoketest-failed, please look into debug output INFO - Cleaning up INFO - Removing disk file: testinstall-20240706-114534-3cf3.img ERROR - Hmm... system got an exception while processing. ERROR - The ISO image is not considered usable! make: *** [Makefile:34: testd] Error 1

dd010101 commented 4 months ago

Seems fine:

test_accel_ipv4_pool (__main__.TestVPNPPTPServer.test_accel_ipv4_pool) ... ok
test_accel_ipv6_pool (__main__.TestVPNPPTPServer.test_accel_ipv6_pool) ... ok
test_accel_limits (__main__.TestVPNPPTPServer.test_accel_limits) ... ok
test_accel_local_authentication (__main__.TestVPNPPTPServer.test_accel_local_authentication) ... ok
test_accel_log_level (__main__.TestVPNPPTPServer.test_accel_log_level) ... ok
test_accel_name_servers (__main__.TestVPNPPTPServer.test_accel_name_servers) ... ok
test_accel_next_pool (__main__.TestVPNPPTPServer.test_accel_next_pool) ... ok
test_accel_ppp_options (__main__.TestVPNPPTPServer.test_accel_ppp_options) ... ok
test_accel_radius_authentication (__main__.TestVPNPPTPServer.test_accel_radius_authentication) ... ok
test_accel_shaper (__main__.TestVPNPPTPServer.test_accel_shaper) ... ok
test_accel_snmp (__main__.TestVPNPPTPServer.test_accel_snmp) ... ok
test_accel_wins_server (__main__.TestVPNPPTPServer.test_accel_wins_server) ... ok

----------------------------------------------------------------------
Ran 12 tests in 67.252s

EDIT: I see you have L2TP test intead of PTTP, that seems also fine:

test_accel_ipv4_pool (__main__.TestVPNL2TPServer.test_accel_ipv4_pool) ... ok
test_accel_ipv6_pool (__main__.TestVPNL2TPServer.test_accel_ipv6_pool) ... ok
test_accel_limits (__main__.TestVPNL2TPServer.test_accel_limits) ... ok
test_accel_local_authentication (__main__.TestVPNL2TPServer.test_accel_local_authentication) ... ok
test_accel_log_level (__main__.TestVPNL2TPServer.test_accel_log_level) ... ok
test_accel_name_servers (__main__.TestVPNL2TPServer.test_accel_name_servers) ... ok
test_accel_next_pool (__main__.TestVPNL2TPServer.test_accel_next_pool) ... ok
test_accel_ppp_options (__main__.TestVPNL2TPServer.test_accel_ppp_options) ... ok
test_accel_radius_authentication (__main__.TestVPNL2TPServer.test_accel_radius_authentication) ... ok
test_accel_shaper (__main__.TestVPNL2TPServer.test_accel_shaper) ... ok
test_accel_snmp (__main__.TestVPNL2TPServer.test_accel_snmp) ... ok
test_accel_wins_server (__main__.TestVPNL2TPServer.test_accel_wins_server) ... ok
test_l2tp_radius_server (__main__.TestVPNL2TPServer.test_l2tp_radius_server) ... ok
test_l2tp_server_authentication_protocols (__main__.TestVPNL2TPServer.test_l2tp_server_authentication_protocols) ... ok
test_vpn_l2tp_dependence_ipsec_swanctl (__main__.TestVPNL2TPServer.test_vpn_l2tp_dependence_ipsec_swanctl) ... ok

----------------------------------------------------------------------
Ran 15 tests in 85.846s

OK

Try to rerun test. The smoketest isn't exactly reliable, there are known issues with race conditions and such.

You may as well ignore the smoketest and do real world tests. The smoketest will not tell you if the image is ok anyway - it will just tell you if something is very wrong but it seems that quite often it will tell you false-positives too... You need to verify the image in real world test anyway - like if it actually applies your config and then it does routes/forwards traffic and works in functional sense.

koljenovic commented 4 months ago

Rerun tests twice, still failing, also rebuilt all the packages in linux-kernel manually (including accel) - that one failed the same tests as well.

dd010101 commented 4 months ago

Try to:

scripts/check-qemu-install --debug --configd build/live-image-amd64.hybrid.iso --sandbox

From the same place where you run make testd from. This launches the same environment as smoketest uses and gives you prompt. It takes a while.

Then you can run the failing test just by itself:

/usr/libexec/vyos/tests/smoke/cli/test_vpn_l2tp.py

The smoketest output shows the path of the script so you can run any test like this, just copy the path from the smoketest.

If you find the test passed by itself then it's false positive since these tests are standalone and thus previous test or the environment shall have no influence over any other test unless the test itself has most likely bug. If the test fails on it's own then it's a bug of VyOS (unlikely) or the test has bug (more likely).

Watch out for OOM killer inside the output of smoketest VM that's also a known issue.

koljenovic commented 4 months ago

Both failed sandboxed as well, I have checked the vyos repo and there were some accel testcases changed/appended around finalizing epa3, that might be it, but what confuses me are your tests passing OK.

dd010101 commented 4 months ago

I don't see any recent changes - https://github.com/vyos/vyos-1x/commits/sagitta/smoketest/scripts/cli/base_accel_ppp_test.py on the test side. What changes you saw?

Anyway... This seems like possible race condition where the test commits and tries to verify that the accel-pppd process is running but there is chance that the test checks for the accel-pppd existence so quick it doesn't give the service chance to start. What happens if you boot the sandbox and add:

import time
time.sleep(3)

Before:

self.assertTrue(process_named_running(self._process_name))

In the def tearDown(self): in file /usr/libexec/vyos/tests/smoke/cli/base_accel_ppp_test.py.

On this line.

So the:

        def tearDown(self):
            # Check for running process
            self.assertTrue(process_named_running(self._process_name))

            self.cli_delete(self._base_path)
            self.cli_commit()

            # Check for running process
            self.assertFalse(process_named_running(self._process_name))

Looks like:

        def tearDown(self):
            import time
            time.sleep(3)
            # Check for running process
            self.assertTrue(process_named_running(self._process_name))

            self.cli_delete(self._base_path)
            self.cli_commit()

            # Check for running process
            self.assertFalse(process_named_running(self._process_name))

And then run the PPTP / L2PP test?

/usr/libexec/vyos/tests/smoke/cli/test_vpn_pptp.py
/usr/libexec/vyos/tests/smoke/cli/test_vpn_l2tp.py
dd010101 commented 1 month ago

Closing due to inactivity. The smoketest isn't just very reliable, that's just how it's written and there were documented randomly failing tests on specific machines without any reason. Likely due to race condition situation where one task completes too slowly or too quickly and the test doesn't expect this. Thus adding delay to problematic parts would likely solve the issue.