kytos-ng / kytos

Kytos SDN Platform. Kytos is designed to be easy to install, use, develop and share Network Apps (NApps).
https://kytos-ng.github.io/
MIT License
2 stars 7 forks source link

feat: parametrized timeout on KytosEventBuffer.put and exposing buffers queue size config #429

Closed viniarck closed 5 months ago

viniarck commented 8 months ago

Closes #423

This PR is on top of #438 since it depends on event_buffer_conf.

Summary

See updated changelog file (it also explains how to use it)

Local Tests

Full KytosEventBuffers counters: {'app': Counter({'kytos/mef_eline.xyz': 1003, 'kytos/topology.port.created': 19, 'kytos/topology.topology_loaded': 1, 'kytos/mef_eline.evcs_loaded': 1})
Kytos couldn't start because of KytosNAppSetupException: NApp kytos/mef_eline exception   Traceback (most recent call last):
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 817, in load_napp
    napp = napp_module.Main(controller=self)
  File "/home/viniarck/repos/kytos/kytos/core/napps/base.py", line 194, in __init__
    self.setup()
  File "/home/viniarck/repos/napps/napps/kytos/mef_eline/main.py", line 72, in setup
    self.send_events(1024)
  File "/home/viniarck/repos/napps/napps/kytos/mef_eline/main.py", line 76, in send_events
    self.controller.buffers.app.put(KytosEvent("kytos/mef_eline.xyz"), timeout=1)
  File "/home/viniarck/repos/kytos/kytos/core/buffers/buffers.py", line 40, in put
    self._queue.sync_q.put(event, timeout=timeout)
  File "/home/viniarck/repos/kytos/.direnv/python-3.9/lib/python3.9/site-packages/janus/__init__.py", line 388, in put
    raise SyncQueueFull
queue.Full

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 263, in start
    self.start_controller()
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 360, in start_controller
    self.load_napps()
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 852, in load_napps
    self.load_napp(napp.username, napp.name)
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 820, in load_napp
    raise KytosNAppSetupException(msg) from exc
kytos.core.exceptions.KytosNAppSetupException: KytosNAppSetupException: NApp kytos/mef_eline exception  

Full KytosEventBuffers counters: {'app': Counter({'kytos/mef_eline.xyz': 1003, 'kytos/topology.port.created': 19, 'kytos/topology.topology_loaded': 1, 'kytos/mef_eline.evcs_loaded': 1})
}
Shutting down Kytos...
[1]    36965 terminated  kytosd -f -n /home/viniarck/repos/napps/napps -E --database mongodb

End-to-End Tests

I'll dispatch e2e execution with this branch and the other NApps PR which are using timeout=1 when putting events during setup():

+ python3 -m pytest tests/ --reruns 2 -r fEr
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.3.0
rootdir: /builds/amlight/kytos-end-to-end-tester/kytos-end-to-end-tests
plugins: rerunfailures-10.2, timeout-2.1.0, anyio-3.6.2
collected 255 items
tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ...................                        [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 23%]
tests/test_e2e_11_mef_eline.py ......                                    [ 26%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X........... [ 45%]
.                                                                        [ 45%]
tests/test_e2e_14_mef_eline.py x                                         [ 46%]
tests/test_e2e_15_mef_eline.py .....                                     [ 48%]
tests/test_e2e_20_flow_manager.py .....................                  [ 56%]
tests/test_e2e_21_flow_manager.py ...                                    [ 57%]
tests/test_e2e_22_flow_manager.py ...............                        [ 63%]
tests/test_e2e_23_flow_manager.py ..............                         [ 69%]
tests/test_e2e_30_of_lldp.py ....                                        [ 70%]
tests/test_e2e_31_of_lldp.py ...                                         [ 71%]
tests/test_e2e_32_of_lldp.py ...                                         [ 72%]
tests/test_e2e_40_sdntrace.py ..............                             [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 81%]
tests/test_e2e_42_sdntrace.py ..                                         [ 82%]
tests/test_e2e_50_maintenance.py ........................                [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                [ 93%]
tests/test_e2e_70_kytos_stats.py ........                                [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                 [100%]
=============================== warnings summary ===============================
../../../../usr/local/lib/python3.9/dist-packages/kytos/core/config.py:235
------------------------------- start/stop times -------------------------------
= 231 passed, 8 skipped, 9 xfailed, 7 xpassed, 1109 warnings in 12224.88s (3:23:44) =