search

kytos-ng / kytos-end-to-end-tests

Kytos End-to-End Tests
0 stars 10 forks source link

Added additional coverage to maintenance e2e tests #282

Closed HeriLFIU closed 4 months ago

HeriLFIU commented 4 months ago

Summary

There was an issue with the filtering of interfaces in maintenance that was not caught by the unit tests nor the e2e tests. With these new added tests similar issues/bugs should be caught by the e2e tests as well.

Local Tests

I just rewrote the unit test code so that it worked as an e2e test.

End-to-End Tests

Attaching to kytos-end-to-end-tests_kytos_1
kytos_1        | Starting enhanced syslogd: rsyslogd.
kytos_1        | /etc/openvswitch/conf.db does not exist ... (warning).
kytos_1        | Creating empty database /etc/openvswitch/conf.db.
kytos_1        | Starting ovsdb-server.
kytos_1        | Configuring Open vSwitch system IDs.
kytos_1        | Starting ovs-vswitchd.
kytos_1        | Enabling remote OVSDB managers.
kytos_1        | + '[' -z '' ']'
kytos_1        | + '[' -z '' ']'
kytos_1        | + echo 'There is no NAPPS_PATH specified. Default will be used.'
kytos_1        | + NAPPS_PATH=
kytos_1        | + sed -i 's/STATS_INTERVAL = 60/STATS_INTERVAL = 7/g' /var/lib/kytos/napps/kytos/of_core/settings.py
kytos_1        | There is no NAPPS_PATH specified. Default will be used.
kytos_1        | + sed -i 's/CONSISTENCY_MIN_VERDICT_INTERVAL =.*/CONSISTENCY_MIN_VERDICT_INTERVAL = 60/g' /var/lib/kytos/napps/kytos/flow_manager/settings.py
kytos_1        | + sed -i 's/LINK_UP_TIMER = 10/LINK_UP_TIMER = 1/g' /var/lib/kytos/napps/kytos/topology/settings.py
kytos_1        | + sed -i 's/DEPLOY_EVCS_INTERVAL = 60/DEPLOY_EVCS_INTERVAL = 5/g' /var/lib/kytos/napps/kytos/mef_eline/settings.py
kytos_1        | + sed -i 's/LLDP_LOOP_ACTIONS = \["log"\]/LLDP_LOOP_ACTIONS = \["disable","log"\]/' /var/lib/kytos/napps/kytos/of_lldp/settings.py
kytos_1        | + sed -i 's/LLDP_IGNORED_LOOPS = {}/LLDP_IGNORED_LOOPS = {"00:00:00:00:00:00:00:01": \[\[4, 5\]\]}/' /var/lib/kytos/napps/kytos/of_lldp/settings.py
kytos_1        | + sed -i 's/CONSISTENCY_COOKIE_IGNORED_RANGE =.*/CONSISTENCY_COOKIE_IGNORED_RANGE = [(0xdd00000000000000, 0xdd00000000000009)]/g' /var/lib/kytos/napps/kytos/flow_manager/settings.py
kytos_1        | + sed -i 's/LIVENESS_DEAD_MULTIPLIER =.*/LIVENESS_DEAD_MULTIPLIER = 3/g' /var/lib/kytos/napps/kytos/of_lldp/settings.py
kytos_1        | + kytosd --help
kytos_1        | + sed -i s/WARNING/INFO/g /etc/kytos/logging.ini
kytos_1        | + test -z ''
kytos_1        | + TESTS=tests/
kytos_1        | + test -z ''
kytos_1        | + RERUNS=2
kytos_1        | + python3 scripts/wait_for_mongo.py
kytos_1        | Trying to run hello command on MongoDB...
kytos_1        | Trying to run 'hello' command on MongoDB...
kytos_1        | Trying to run 'hello' command on MongoDB...
kytos_1        | Ran 'hello' command on MongoDB successfully. It's ready!
kytos_1        | + python3 -m pytest --timeout=60 tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_135_interface_payload_filtering
kytos_1        | ============================= test session starts ==============================
kytos_1        | platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.4.0
kytos_1        | rootdir: /tests
kytos_1        | plugins: timeout-2.1.0, rerunfailures-10.2, anyio-3.6.2
kytos_1        | timeout: 60.0s
kytos_1        | timeout method: signal
kytos_1        | timeout func_only: False
kytos_1        | collected 1 item
kytos_1        | 
kytos_1        | tests/test_e2e_50_maintenance.py F                                       [100%]
kytos_1        | 
kytos_1        | =================================== FAILURES ===================================
kytos_1        | ___________ TestE2EMaintenance.test_135_interface_payload_filtering ____________
kytos_1        | 
kytos_1        | self = <tests.test_e2e_50_maintenance.TestE2EMaintenance object at 0x7f2e33434df0>
kytos_1        | 
kytos_1        |     def test_135_interface_payload_filtering(self):
kytos_1        |         self.net.start_controller(clean_config=True, enable_all=True)
kytos_1        |         self.net.wait_switches_connect()
kytos_1        |         time.sleep(10)
kytos_1        |     
kytos_1        |         start = datetime.now(pytz.utc) + timedelta(days=1)
kytos_1        |         end = start + timedelta(hours=2)
kytos_1        |         payload = {
kytos_1        |             "start": start.strftime(TIME_FMT),
kytos_1        |             "end": end.strftime(TIME_FMT),
kytos_1        |             "interfaces": [
kytos_1        |                 "00:00:00:00:00:00:00:01:1",
kytos_1        |             ],
kytos_1        |         }
kytos_1        |         api_url = KYTOS_API + '/maintenance/v1'
kytos_1        |         response = requests.post(api_url, data=json.dumps(payload), headers={'Content-type': 'application/json'})
kytos_1        |         data = response.json()
kytos_1        | >       assert response.status_code == 201, data
kytos_1        | E       AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['00:00:00:00:00:00:00:01:1']}"}
kytos_1        | E       assert 400 == 201
kytos_1        | E        +  where 400 = <Response [400]>.status_code
kytos_1        | 
kytos_1        | tests/test_e2e_50_maintenance.py:1303: AssertionError
kytos_1        | ---------------------------- Captured stdout setup -----------------------------
kytos_1        | FAIL to stop kytos after 5 seconds -- Kytos pid still exists.. Force stop!
kytos_1        | ---------------------------- Captured stderr setup -----------------------------
kytos_1        | *** Error setting resource limits. Mininet's performance may be affected.
kytos_1        | Unable to contact the remote controller at 127.0.0.1:6653
kytos_1        | ------------------------------ Captured log setup ------------------------------
kytos_1        | WARNING  mininet:log.py:153 *** Error setting resource limits. Mininet's performance may be affected.
kytos_1        | 
kytos_1        | WARNING  mininet:log.py:153 Unable to contact the remote controller at 127.0.0.1:6653
kytos_1        | ----------------------------- Captured stdout call -----------------------------
kytos_1        | FAIL to stop kytos after 5 seconds -- Kytos pid still exists.. Force stop!
kytos_1        | =============================== warnings summary ===============================
kytos_1        | test_e2e_50_maintenance.py: 17 warnings
kytos_1        |   /usr/lib/python3/dist-packages/mininet/node.py:1121: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
kytos_1        |     return ( StrictVersion( cls.OVSVersion ) <
kytos_1        | 
kytos_1        | test_e2e_50_maintenance.py: 17 warnings
kytos_1        |   /usr/lib/python3/dist-packages/mininet/node.py:1122: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
kytos_1        |     StrictVersion( '1.10' ) )
kytos_1        | 
kytos_1        | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
kytos_1        | ------------------------------- start/stop times -------------------------------
kytos_1        | test_e2e_50_maintenance.py::TestE2EMaintenance::test_135_interface_payload_filtering: 2024-02-28,22:24:58.660133 - 2024-02-28,22:25:16.144194
kytos_1        | =========================== short test summary info ============================
kytos_1        | FAILED tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_135_interface_payload_filtering
kytos_1        | ======================= 1 failed, 34 warnings in 51.06s ========================
kytos-end-to-end-tests_kytos_1 exited with code 1

The new test seems to function properly and detects the previous bug.

HeriLFIU commented 4 months ago

@viniarck I don't know if this is necessarily what I had to do, but the e2e tests can now also detect the bug, so they have increased coverage.

HeriLFIU commented 4 months ago

I just rewrote the 4 unit tests into e2e tests, so I don't know if that is fine for this case.

HeriLFIU commented 4 months ago

@viniarck I just removed it and replaced it with datetime.utcnow()

viniarck commented 4 months ago

@viniarck I just removed it and replaced it with datetime.utcnow()

Much appreciated.

@HeriLFIU, the new tests cases are failing though, can you double check if the link id you used is correct? 

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
------------------------------- start/stop times -------------------------------
rerun: 0
tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering: 2024-02-29,17:44:31.646229 - 2024-02-29,17:44:55.036919
self = <tests.test_e2e_50_maintenance.TestE2EMaintenance object at 0x7f3d0a8804f0>
    def test_125_multiple_payload_item_filtering(self):
        self.net.start_controller(clean_config=True, enable_all=True)
        self.net.wait_switches_connect()
        time.sleep(10)

        start = datetime.now(pytz.utc) + timedelta(days=1)
        end = start + timedelta(hours=2)
        payload = {
            "start": start.strftime(TIME_FMT),
            "end": end.strftime(TIME_FMT),
            "switches": ["00:00:00:00:00:00:00:02", "00:00:00:00:00:00:00:03"],
            "interfaces": ["00:00:00:00:00:00:00:03:3", "00:00:00:00:00:00:00:02:1"],
            "links": [
                "cf0f4071be[426](https://gitlab.ampath.net/amlight/kytos-end-to-end-tester/-/jobs/59329#L426)b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260",
                "4d42dc0852278accac7d9df15418f6d921db160b13d674029a87cef1b5f67f30",
            ],
        }
        api_url = KYTOS_API + '/maintenance/v1'
        response = requests.post(api_url, data=json.dumps(payload), headers={'Content-type': 'application/json'})
        data = response.json()
>       assert response.status_code == 201, data
E       AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260']}"}
E       assert 400 == 201
E        +  where 400 = <Response [400]>.status_code
tests/test_e2e_50_maintenance.py:1248: AssertionError
rerun: 1
tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering: 2024-02-29,17:44:55.393942 - 2024-02-29,17:45:30.252966
self = <tests.test_e2e_50_maintenance.TestE2EMaintenance object at 0x7f3d0a8804f0>
    def test_125_multiple_payload_item_filtering(self):
        self.net.start_controller(clean_config=True, enable_all=True)
        self.net.wait_switches_connect()
        time.sleep(10)

        start = datetime.now(pytz.utc) + timedelta(days=1)
        end = start + timedelta(hours=2)
        payload = {
            "start": start.strftime(TIME_FMT),
            "end": end.strftime(TIME_FMT),
            "switches": ["00:00:00:00:00:00:00:02", "00:00:00:00:00:00:00:03"],
            "interfaces": ["00:00:00:00:00:00:00:03:3", "00:00:00:00:00:00:00:02:1"],
            "links": [
                "cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260",
                "4d42dc0852278accac7d9df15418f6d921db160b13d674029a87cef1b5f67f30",
            ],
        }
        api_url = KYTOS_API + '/maintenance/v1'
        response = requests.post(api_url, data=json.dumps(payload), headers={'Content-type': 'application/json'})
        data = response.json()
>       assert response.status_code == 201, data
E       AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260']}"}
E       assert 400 == 201
E        +  where 400 = <Response [400]>.status_code
tests/test_e2e_50_maintenance.py:1248: AssertionError
rerun: 0
tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering: 2024-02-29,17:47:05.767004 - 2024-02-29,17:47:40.052241
self = <tests.test_e2e_50_maintenance.TestE2EMaintenance object at 0x7f3d0a880100>
    def test_140_link_payload_filtering(self):
        self.net.start_controller(clean_config=True, enable_all=True)
        self.net.wait_switches_connect()
        time.sleep(10)

        start = datetime.now(pytz.utc) + timedelta(days=1)
        end = start + timedelta(hours=2)
        payload = {
            "start": start.strftime(TIME_FMT),
            "end": end.strftime(TIME_FMT),
            "links": [
                "cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260",
            ],
        }
        api_url = KYTOS_API + '/maintenance/v1'
        response = requests.post(api_url, data=json.dumps(payload), headers={'Content-type': 'application/json'})
        data = response.json()
>       assert response.status_code == 201, data
E       AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260']}"}
E       assert 400 == 201
E        +  where 400 = <Response [400]>.status_code
tests/test_e2e_50_maintenance.py:1341: AssertionError
rerun: 1
tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering: 2024-02-29,17:48:28.200069 - 2024-02-29,17:48:52.695713
self = <tests.test_e2e_50_maintenance.TestE2EMaintenance object at 0x7f3d0a880100>
    def test_140_link_payload_filtering(self):
        self.net.start_controller(clean_config=True, enable_all=True)
        self.net.wait_switches_connect()
        time.sleep(10)

        start = datetime.now(pytz.utc) + timedelta(days=1)
        end = start + timedelta(hours=2)
        payload = {
            "start": start.strftime(TIME_FMT),
            "end": end.strftime(TIME_FMT),
            "links": [
                "cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260",
            ],
        }
        api_url = KYTOS_API + '/maintenance/v1'
        response = requests.post(api_url, data=json.dumps(payload), headers={'Content-type': 'application/json'})
        data = response.json()
>       assert response.status_code == 201, data
E       AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['cf0f4071be426b3f745027f5d22bc61f8312ae86293c9b28e7e66015607a9260']}"}
E       assert 400 == 201
E        +  where 400 = <Response [400]>.status_code
tests/test_e2e_50_maintenance.py:1341: AssertionError
tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering: 2024-02-29,17:45:30.[436](https://gitlab.ampath.net/amlight/kytos-end-to-end-tester/-/jobs/59329#L436)958 - 2024-02-29,17:46:05.720633
tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering: 2024-02-29,17:49:41.767493 - 2024-02-29,17:50:06.348797
=========================== rerun test summary info ============================
RERUN tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering
RERUN tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering
RERUN tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering
RERUN tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering
=========================== short test summary info ============================
FAILED tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering - AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['cf0f4071be426b3f7[450](https://gitlab.ampath.net/amlight/kytos-end-to-end-tester/-/jobs/59329#L450)27f5d22bc61f8312ae86293c9b28e7e66015607a9260']}"}
assert 400 == 201
 +  where 400 = <Response [400]>.status_code
FAILED tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering - AssertionError: {'code': 400, 'description': "Window contains non-existant items: {'switches': [], 'interfaces': [], 'links': ['cf0f4071be426b3f74[502](https://gitlab.ampath.net/amlight/kytos-end-to-end-tester/-/jobs/59329#L502)7f5d22bc61f8312ae86293c9b28e7e66015607a9260']}"}
assert 400 == 201
 +  where 400 = <Response [400]>.status_code
= 2 failed, 235 passed, 8 skipped, 9 xfailed, 7 xpassed, 1211 warnings, 4 rerun in 13[523](https://gitlab.ampath.net/amlight/kytos-end-to-end-tester/-/jobs/59329#L523).79s (3:45:23) =
HeriLFIU commented 4 months ago

@viniarck Should be fixed. Mb, I hadn't run the local tests with the bug fix implemented to make sure everything was running properly. For some reason the topology generated during the tests have different link IDs from the topologys generated locally.

(kn) root@heriberto-VirtualBox:/home/heriberto/kytos-project/kytos-end-to-end-tests# python3 -m pytest tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_140_link_payload_filtering
======================================================================== test session starts ========================================================================
platform linux -- Python 3.9.18, pytest-7.2.1, pluggy-1.3.0
rootdir: /home/heriberto/kytos-project/kytos-end-to-end-tests
plugins: timeout-2.2.0, asyncio-0.20.3, cov-4.0.0, anyio-3.6.2, rerunfailures-13.0
asyncio: mode=strict
collected 1 item                                                                                                                                                    

tests/test_e2e_50_maintenance.py .                                                                                                                            [100%]

========================================================================= warnings summary ==========================================================================
tests/test_e2e_50_maintenance.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1106: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    return ( StrictVersion( cls.OVSVersion ) <

tests/test_e2e_50_maintenance.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1107: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    StrictVersion( '1.10' ) )

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
------------------------------------------------------------------------- start/stop times --------------------------------------------------------------------------
================================================================== 1 passed, 34 warnings in 51.62s ==================================================================
HeriLFIU commented 4 months ago

Test 125 is also now passing.

(kn) root@heriberto-VirtualBox:/home/heriberto/kytos-project/kytos-end-to-end-tests# python3 -m pytest tests/test_e2e_50_maintenance.py::TestE2EMaintenance::test_125_multiple_payload_item_filtering
======================================================================== test session starts ========================================================================
platform linux -- Python 3.9.18, pytest-7.2.1, pluggy-1.3.0
rootdir: /home/heriberto/kytos-project/kytos-end-to-end-tests
plugins: timeout-2.2.0, asyncio-0.20.3, cov-4.0.0, anyio-3.6.2, rerunfailures-13.0
asyncio: mode=strict
collected 1 item                                                                                                                                                    

tests/test_e2e_50_maintenance.py .                                                                                                                            [100%]

========================================================================= warnings summary ==========================================================================
tests/test_e2e_50_maintenance.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1106: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    return ( StrictVersion( cls.OVSVersion ) <

tests/test_e2e_50_maintenance.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1107: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    StrictVersion( '1.10' ) )

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
------------------------------------------------------------------------- start/stop times --------------------------------------------------------------------------
================================================================== 1 passed, 34 warnings in 51.74s ==================================================================
viniarck commented 4 months ago

Cool. Let's ship this @HeriLFIU, nicely done.