search

kytos-ng / kytos-end-to-end-tests

Kytos End-to-End Tests
0 stars 10 forks source link

`test_e2e_40_sdntrace.py` has been failing over the past days #255

Closed viniarck closed 11 months ago

viniarck commented 11 months ago

It might be related to issue https://github.com/kytos-ng/kytos-end-to-end-tests/issues/239, but not it's failing even more than usual. @gretelliz if you can take care of this, it's about time to make this test suite reliable and properly allocating and deallocating the resources between test cases. We can't afford to have flaky e2e test cases especially during the finish line of a release.

list_results = data["result"]
>       assert len(list_results[0]) == 3
E       AssertionError: assert 2 == 3
E        +  where 2 = len([{'dpid': '00:00:00:00:00:00:00:01', 'port': 2, 'time': '2023-08-08 06:28:54.388184', 'type': 'starting', ...}, {'dpid': '00:00:00:00:00:00:00:02', 'out': {'port': 3, 'vlan': 100}, 'port': 2, 'time': '2023-08-08 06:28:54.388249', ...}])
tests/test_e2e_40_sdntrace.py:544: AssertionError
tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow: 2023-08-08,06:27:39.152722 - 2023-08-08,06:27:49.211854
tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows: 2023-08-08,06:27:53.546585 - 2023-08-08,06:27:53.562754
tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action: 2023-08-08,06:28:54.481320 - 2023-08-08,06:29:24.669314
=========================== rerun test summary info ============================
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action
=========================== short test summary info ============================
FAILED tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow - AssertionError: {'result': [{'dpid': '00:00:00:00:00:00:00:01', 'port': 1, 'time': '2023-08-08 06:27:49.208860', 'type': 'starting', 'vlan': 400}, {'dpid': '00:00:00:00:00:00:00:02', 'port': 2, 'time': '2023-08-08 06:27:49.208930', 'type': 'intermediary', 'vlan': 1}, {'dpid': '00:00:00:00:00:00:00:03', 'port': 2, 'time': '2023-08-08 06:27:49.208958', 'type': 'intermediary', 'vlan': 1}, {'dpid': '00:00:00:00:00:00:00:04', 'port': 2, 'time': '2023-08-08 06:27:49.208982', 'type': 'intermediary', 'vlan': 1, 'out': {'port': 3, 'vlan': 1}}]}
assert 4 == 5
 +  where 4 = len([{'dpid': '00:00:00:00:00:00:00:01', 'port': 1, 'time': '2023-08-08 06:27:49.208[860](https://gitlab.ampath.net/kytos/kytos-end-to-end-tester/-/jobs/48989#L860)', 'type': 'starting', ...}, {'dpid...dpid': '00:00:00:00:00:00:00:04', 'out': {'port': 3, 'vlan': 1}, 'port': 2, 'time': '2023-08-08 06:27:49.208982', ...}])
FAILED tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows - AssertionError: {"description":"The EVC already exists.","code":409}
assert 409 == 201
 +  where 409 = <Response [409]>.status_code
FAILED tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action - AssertionError: assert 2 == 3
 +  where 2 = len([{'dpid': '00:00:00:00:00:00:00:01', 'port': 2, 'time': '2023-08-08 06:29:24.667055', 'type': 'starting', ...}, {'dpid': '00:00:00:00:00:00:00:02', 'out': {'port': 3, 'vlan': 100}, 'port': 2, 'time': '2023-08-08 06:29:24.667120', ...}])
= 3 failed, 214 passed, 6 skipped, 11 xfailed, 5 xpassed, [867](https://gitlab.ampath.net/kytos/kytos-end-to-end-tester/-/jobs/48989#L867) warnings, 6 rerun in 11003.79s (3:03:23) =
Cleaning up file based variables
>       assert len(list_results[0]) == 3
E       AssertionError: assert 2 == 3
E        +  where 2 = len([{'dpid': '00:00:00:00:00:00:00:01', 'port': 2, 'time': '2023-08-03 06:29:18.835342', 'type': 'starting', ...}, {'dpid': '00:00:00:00:00:00:00:02', 'out': {'port': 3, 'vlan': 100}, 'port': 2, 'time': '2023-08-03 06:29:18.835412', ...}])
tests/test_e2e_40_sdntrace.py:544: AssertionError
tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow: 2023-08-03,06:28:03.660488 - 2023-08-03,06:28:13.792316
tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows: 2023-08-03,06:28:18.391640 - 2023-08-03,06:28:18.410126
tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action: 2023-08-03,06:29:18.953645 - 2023-08-03,06:29:49.140101
=========================== rerun test summary info ============================
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action
RERUN tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action
=========================== short test summary info ============================
FAILED tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_020_run_sdntrace_fail_missing_flow - AssertionError: {'result': [{'dpid': '00:00:00:00:00:00:00:01', 'port': 1, 'time': '2023-08-03 06:28:13.789370', 'type': 'starting', 'vlan': 400}, {'dpid': '00:00:00:00:00:00:00:02', 'port': 2, 'time': '2023-08-03 06:28:13.789431', 'type': 'intermediary', 'vlan': 1}, {'dpid': '00:00:00:00:00:00:00:03', 'port': 2, 'time': '2023-08-03 06:28:13.789456', 'type': 'intermediary', 'vlan': 1}, {'dpid': '00:00:00:00:00:00:00:04', 'port': 2, 'time': '2023-08-03 06:28:13.789481', 'type': 'intermediary', 'vlan': 1, 'out': {'port': 3, 'vlan': 1}}]}
assert 4 == 5
 +  where 4 = len([{'dpid': '00:00:00:00:00:00:00:01', 'port': 1, 'time': '2023-08-03 06:28:13.789370', 'type': 'starting', ...}, {'dpid...dpid': '00:00:00:00:00:00:00:04', 'out': {'port': 3, 'vlan': 1}, 'port': 2, 'time': '2023-08-03 06:28:13.789481', ...}])
FAILED tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_030_run_sdntrace_for_stored_flows - AssertionError: {"description":"The EVC already exists.","code":409}
assert 409 == 201
 +  where 409 = <Response [409]>.status_code
FAILED tests/test_e2e_40_sdntrace.py::TestE2ESDNTrace::test_040_run_sdntrace_no_action - AssertionError: assert 2 == 3
 +  where 2 = len([{'dpid': '00:00:00:00:00:00:00:01', 'port': 2, 'time': '2023-08-03 06:29:49.137458', 'type': 'starting', ...}, {'dpid': '00:00:00:00:00:00:00:02', 'out': {'port': 3, 'vlan': 100}, 'port': 2, 'time': '2023-08-03 06:29:49.137527', ...}])
= 3 failed, 214 passed, 6 skipped, 11 xfailed, 5 xpassed, [867](https://gitlab.ampath.net/kytos/kytos-end-to-end-tester/-/jobs/48721#L867) warnings, 6 rerun in 11044.01s (3:04:04) =