search

kytos-ng / kytos-end-to-end-tests

Kytos End-to-End Tests
0 stars 10 forks source link

Added the functionality of being able to run tests locally #271

Closed HeriLFIU closed 5 months ago

HeriLFIU commented 7 months ago

Closes #238

Summary

You can now run tests locally.

Setup/Instructions

_Most setup is found within the local_setup folder._

  1. You can start running tests locally by adding the mongoLT (Local Test) hosts with the add-etc-local-hosts.sh bash script.
  2. Then you can add the required environment variables with the add-persistent-env-variables.sh bash script.
  3. Subsequently, the docker-compose.local.yml file can be used with the command sudo docker-compose -f docker-compose.local.yml up -d to run all of the required docker containers.
  4. Switch to root user with sudo su - as mininet will only run as root and when sudo is used it doesn't have the required dependencies to run kytos and the tests.

    Bugs Fixed

    FAILED tests/test_e2e_12_mef_eline.py::TestE2EMefEline::test_create_schedule_by_date - AssertionError: assert False is True Changed datetime.now() to datetime.utcnow() since datetime.now() would return the EST time and then switch the time zone to UTC while keeping the original values, causing the job to be terminated since it should have occurred 5 hours ago.

    Current Issues

    Tests are running locally but they seem to be delayed due to the limited resources of my VM. This increases the time to process and generate things such as EVCs drastically by around 40 to 50 seconds, which then requires longer wait/sleep times to compensate.

HeriLFIU commented 7 months ago

It was a bit tricky but for rs-init.sh I found a very simple fix. I iterated through the string of MONGO_NODES and added them to an array, which allowed me to keep almost all of the original code/functionality of the bash script.

viniarck commented 7 months ago

Good news @HeriLFIU, the entire topology test suite passed locally without running kytosd in a container, I just had to adapt --timeout=180, so no need to change how kytosd is being pkill'ed:

[vx kytos-end-to-end-tests]# python3 -m pytest --timeout=180 tests -k topology
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.9.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/viniarck/repos/kytos-end-to-end-tests
plugins: anyio-3.6.2, timeout-2.2.0
timeout: 180.0s
timeout method: signal
timeout func_only: False
collected 254 items / 236 deselected / 18 selected                                                                                                                                      

tests/test_e2e_05_topology.py ..................                                                                                                                                  [100%]

=================================================================================== warnings summary ====================================================================================
../kytos/kytos/core/config.py:186
  /home/viniarck/repos/kytos/kytos/core/config.py:186: UserWarning: Unknown arguments: ['--timeout=180', 'tests', '-k', 'topology']
    warnings.warn(f"Unknown arguments: {unknown}")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
----------------------------------------------------------------------------------- start/stop times ------------------------------------------------------------------------------------
=============================================================== 18 passed, 236 deselected, 1 warning in 842.58s (0:14:02) ===============================================================

When I have the chance, I'll leave the entire test suite running. Let's make sure to also address the other threads that I opened, and then once all test suites are passing we can land this PR.

viniarck commented 7 months ago

When running all tests suites, I ended up hitting other test failures. Overall, at least the setup to run it locally it's progressing well, so now it's matter of analyzing and sorting, and stabilizing any of these potential failures. Ideally, we shouldn't have any surprises we since we know for a fact that on GitLab host e2e nightly tests has been stable, so it might also be any potential local change that I had or something else. Either way, it needs to be explored more locally, and then we can make a decision how it'll be stabilized and how this will land. 

[vx kytos-end-to-end-tests]# python3 -m pytest --timeout=180 tests 
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.9.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/viniarck/repos/kytos-end-to-end-tests
plugins: anyio-3.6.2, timeout-2.2.0
timeout: 180.0s
timeout method: signal
timeout func_only: False
collected 254 items                                                                                                                                                                     

tests/test_e2e_01_kytos_startup.py FF                                                                                                                                             [  0%]
tests/test_e2e_05_topology.py ..................                                                                                                                                  [  7%]
tests/test_e2e_10_mef_eline.py ..........ssF....x.....x........F.......                                                                                                           [ 23%]
tests/test_e2e_11_mef_eline.py ......                                                                                                                                             [ 25%]
tests/test_e2e_12_mef_eline.py .....Xx.                                                                                                                                           [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X............                                                                                                         [ 45%]
tests/test_e2e_14_mef_eline.py x                                                                                                                                                  [ 46%]
tests/test_e2e_15_mef_eline.py .....                                                                                                                                              [ 48%]
tests/test_e2e_20_flow_manager.py ......FFFF...........                                                                                                                           [ 56%]
tests/test_e2e_21_flow_manager.py ...                                                                                                                                             [ 57%]
tests/test_e2e_22_flow_manager.py ...............                                                                                                                                 [ 63%]
tests/test_e2e_23_flow_manager.py ..............                                                                                                                                  [ 68%]
tests/test_e2e_30_of_lldp.py .F..                                                                                                                                                 [ 70%]
tests/test_e2e_31_of_lldp.py ..F                                                                                                                                                  [ 71%]
tests/test_e2e_32_of_lldp.py F.F                                                                                                                                                  [ 72%]
tests/test_e2e_40_sdntrace.py ..............                                                                                                                                      [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                                                                                                                          [ 81%]
tests/test_e2e_42_sdntrace.py ..                                                                                                                                                  [ 82%]
tests/test_e2e_50_maintenance.py .F....F.FFFFFFF.F.FFF.FF                                                                                                                         [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                                                                                                                         [ 93%]
tests/test_e2e_70_kytos_stats.py ....FF..                                                                                                                                         [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                                                                                                                          [100%]
HeriLFIU commented 6 months ago

I was trying to fix some of the failed tests on maintenance since there was a big chunk of them there when then I realized some of them were due to the datetime.now() error. @viniarck

HeriLFIU commented 6 months ago

@viniarck 

(kn) root@heriberto-VirtualBox:/home/heriberto/kytos-project/kytos-end-to-end-tests# pytest tests/test_e2e_50_maintenance.py
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/heriberto/kytos-project/kytos-end-to-end-tests
plugins: anyio-3.6.2
collected 24 items                                                                                                                                                      

tests/test_e2e_50_maintenance.py ........................                                                                                                         [100%]

=========================================================================== warnings summary ============================================================================
tests/test_e2e_50_maintenance.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1106: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    return ( StrictVersion( cls.OVSVersion ) <

tests/test_e2e_50_maintenance.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1107: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    StrictVersion( '1.10' ) )

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
--------------------------------------------------------------------------- start/stop times ----------------------------------------------------------------------------
============================================================= 24 passed, 34 warnings in 1445.27s (0:24:05) ==============================================================

Maintenance e2e tests should be passing, it was all just datetime.now(). I replaced them with datetime.utcnow() and it should be working now.

HeriLFIU commented 6 months ago 
(kn) root@heriberto-VirtualBox:/home/heriberto/kytos-project/kytos-end-to-end-tests# pytest tests/test_e2e_70_kytos_stats.py
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/heriberto/kytos-project/kytos-end-to-end-tests
plugins: anyio-3.6.2
collected 8 items                                                                                                                                                       

tests/test_e2e_70_kytos_stats.py ........                                                                                                                         [100%]

=========================================================================== warnings summary ============================================================================
tests/test_e2e_70_kytos_stats.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1106: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    return ( StrictVersion( cls.OVSVersion ) <

tests/test_e2e_70_kytos_stats.py: 17 warnings
  /home/heriberto/kytos-project/kn/lib/python3.9/site-packages/mininet/node.py:1107: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    StrictVersion( '1.10' ) )

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
--------------------------------------------------------------------------- start/stop times ----------------------------------------------------------------------------
============================================================== 8 passed, 34 warnings in 498.76s (0:08:18) ===============================================================

For Kytos stats I encountered the same issue I had with some of the other tests, at first some of them were failing, but after increasing the wait times by around 40 seconds they all passed.

HeriLFIU commented 6 months ago

One neat thing that I realized I can do. It was probably obvious, but I just noticed. Since the tests are running locally I can add my own logs and other additions in the code, such as the napps I am debugging.

HeriLFIU commented 5 months ago

@viniarck I'm running all of the tests now to see if they are all functioning properly. I also made some changes to kytos-init.sh. It'll first check for an environment variable called NAPPS_PATH. If NAPPS_PATH is not found it will then use the find command to search for the matching default NApps path ('var/lib/kytos/napps/') within the project folder, and if neither are found it'll just go with the default path of 'var/lib/kytos/napps/'.

HeriLFIU commented 5 months ago

They are passing

+ python3 -m pytest tests/ --reruns 2 -r fEr
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/heriberto/kytos-project/kytos-end-to-end-tests
plugins: timeout-2.2.0, anyio-3.6.2, rerunfailures-13.0
collected 261 items                                                                                                                                                     

tests/test_e2e_01_kytos_startup.py ..                                                                                                                             [  0%]
tests/test_e2e_05_topology.py ...................                                                                                                                 [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................                                                                                           [ 23%]
tests/test_e2e_11_mef_eline.py ......                                                                                                                             [ 25%]
tests/test_e2e_12_mef_eline.py .....Xx.                                                                                                                           [ 28%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X............                                                                                         [ 44%]
tests/test_e2e_14_mef_eline.py x                                                                                                                                  [ 45%]
tests/test_e2e_15_mef_eline.py .....                                                                                                                              [ 47%]
tests/test_e2e_20_flow_manager.py ........R.............                                                                                                          [ 55%]
tests/test_e2e_21_flow_manager.py ...                                                                                                                             [ 56%]
tests/test_e2e_22_flow_manager.py ...............                                                                                                                 [ 62%]
tests/test_e2e_23_flow_manager.py ..............                                                                                                                  [ 67%]
tests/test_e2e_30_of_lldp.py .R...                                                                                                                                [ 68%]
tests/test_e2e_31_of_lldp.py ...                                                                                                                                  [ 70%]
tests/test_e2e_32_of_lldp.py ...                                                                                                                                  [ 71%]
tests/test_e2e_40_sdntrace.py ..............                                                                                                                      [ 76%]
tests/test_e2e_41_kytos_auth.py ........                                                                                                                          [ 79%]
tests/test_e2e_42_sdntrace.py ..                                                                                                                                  [ 80%]
tests/test_e2e_50_maintenance.py ........................                                                                                                         [ 89%]
tests/test_e2e_60_of_multi_table.py .....                                                                                                                         [ 91%]
tests/test_e2e_70_kytos_stats.py ........                                                                                                                         [ 94%]
tests/test_e2e_80_pathfinder.py ss......                                                                                                                          [ 97%]

and ====================================== 231 passed, 8 skipped, 9 xfailed, 7 xpassed, 1143 warnings, 2 rerun in 11293.06s (3:08:13) =======================================

viniarck commented 5 months ago

@HeriLFIU, it's also passing on GitLab on docker as expected (so, we don't expect e2e nightly tests to fail). This PR will get merged by today:

+ echo 'There is no NAPPS_PATH specified. Default will be used.'
There is no NAPPS_PATH specified. Default will be used.
+ NAPPS_PATH=
+ sed -i 's/STATS_INTERVAL = 60/STATS_INTERVAL = 7/g' /var/lib/kytos/napps/kytos/of_core/settings.py
+ sed -i 's/CONSISTENCY_MIN_VERDICT_INTERVAL =.*/CONSISTENCY_MIN_VERDICT_INTERVAL = 60/g' /var/lib/kytos/napps/kytos/flow_manager/settings.py
+ sed -i 's/LINK_UP_TIMER = 10/LINK_UP_TIMER = 1/g' /var/lib/kytos/napps/kytos/topology/settings.py
+ sed -i 's/DEPLOY_EVCS_INTERVAL = 60/DEPLOY_EVCS_INTERVAL = 5/g' /var/lib/kytos/napps/kytos/mef_eline/settings.py
+ sed -i 's/LLDP_LOOP_ACTIONS = \["log"\]/LLDP_LOOP_ACTIONS = \["disable","log"\]/' /var/lib/kytos/napps/kytos/of_lldp/settings.py
+ sed -i 's/LLDP_IGNORED_LOOPS = {}/LLDP_IGNORED_LOOPS = {"00:00:00:00:00:00:00:01": \[\[4, 5\]\]}/' /var/lib/kytos/napps/kytos/of_lldp/settings.py
+ sed -i 's/CONSISTENCY_COOKIE_IGNORED_RANGE =.*/CONSISTENCY_COOKIE_IGNORED_RANGE = [(0xdd00000000000000, 0xdd00000000000009)]/g' /var/lib/kytos/napps/kytos/flow_manager/settings.py
+ sed -i 's/LIVENESS_DEAD_MULTIPLIER =.*/LIVENESS_DEAD_MULTIPLIER = 3/g' /var/lib/kytos/napps/kytos/of_lldp/settings.py
+ kytosd --help
+ sed -i s/WARNING/INFO/g /etc/kytos/logging.ini
+ test -z tests/
+ test -z ''
+ RERUNS=2
+ python3 scripts/wait_for_mongo.py
Trying to run hello command on MongoDB...
Trying to run 'hello' command on MongoDB...
Ran 'hello' command on MongoDB successfully. It's ready!
+ python3 -m pytest tests/ --reruns 2 -r fEr
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.3.0
rootdir: /builds/amlight/kytos-end-to-end-tester/kytos-end-to-end-tests
plugins: rerunfailures-10.2, timeout-2.1.0, anyio-3.6.2
collected [255](https://gitlab.ampath.net/amlight/kytos-end-to-end-tester/-/jobs/57055#L255) items
tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ...................                        [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 23%]
tests/test_e2e_11_mef_eline.py ......                                    [ 26%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X........... [ 45%]
.                                                                        [ 45%]
tests/test_e2e_14_mef_eline.py x                                         [ 46%]
tests/test_e2e_15_mef_eline.py .....                                     [ 48%]
tests/test_e2e_20_flow_manager.py .....................                  [ 56%]
tests/test_e2e_21_flow_manager.py ...                                    [ 57%]
tests/test_e2e_22_flow_manager.py ...............                        [ 63%]
tests/test_e2e_23_flow_manager.py ..............                         [ 69%]
tests/test_e2e_30_of_lldp.py ....                                        [ 70%]
tests/test_e2e_31_of_lldp.py ...                                         [ 71%]
tests/test_e2e_32_of_lldp.py ...                                         [ 72%]
tests/test_e2e_40_sdntrace.py ..............                             [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 81%]
tests/test_e2e_42_sdntrace.py ..                                         [ 82%]
tests/test_e2e_50_maintenance.py ........................                [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                [ 93%]
tests/test_e2e_70_kytos_stats.py ........                                [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                 [100%]
=============================== warnings summary ===============================
------------------------------- start/stop times -------------------------------
= 231 passed, 8 skipped, 9 xfailed, 7 xpassed, 1109 warnings in 12083.53s (3:21:23) =
HeriLFIU commented 5 months ago

@viniarck I also just added the logging explanation.

viniarck commented 5 months ago

@viniarck I also just added the logging explanation.

Great stuff, it'll be very helpful for other developers. Let's ship this.