kytos-ng / pathfinder

Kytos main path finder Network Application (NApp)
https://kytos-ng.github.io/api/pathfinder.html
MIT License
0 stars 7 forks source link

Thread issue when links metadata has been updated but a new link hasn't been published yet #33

Closed viniarck closed 1 year ago

viniarck commented 1 year ago

I managed to catch the underlying issue that results in this failure https://github.com/amlight/kytos-end-to-end-tests/issues/179.

I turns out that topology when creating a new link, a new request adding or removing metadata can succeed, but kytos.topology.updated hasn't been announced yet. For instance, here's the issue being reproduced in slow motion with sleeps to facilitate to demonstrate the issue:

kytos $> 2022-11-03 16:58:01,392 - INFO [kytos.napps.kytos/topology] [main.py:884:add_links] (thread_pool_app_1) slow sleep...
2022-11-03 16:58:01,395 - INFO [kytos.napps.kytos/topology] [main.py:884:add_links] (thread_pool_app_4) slow sleep...
2022-11-03 16:58:01,397 - INFO [kytos.napps.kytos/topology] [main.py:884:add_links] (thread_pool_app_2) slow sleep...
kytos $>

kytos $>

kytos $> 2022-11-03 16:58:04,552 - INFO [werkzeug] [_internal.py:225:_log] (Thread-97) 127.0.0.1 - - [03/Nov/2022 16:58:04] "POST /api/kytos/topology/v3/links/4d42dc0852278accac7d9df154
18f6d921db160b13d674029a87cef1b5f67f30/metadata HTTP/1.1" 201 -
2022-11-03 16:58:04,576 - ERROR [kytos.core.helpers] [helpers.py:151:handler_context_apm] (thread_pool_app_5) listen_to handler: <function Main.on_links_metadata_changed at 0x7f159820b3
a0>, args: (<Main(pathfinder, stopped 139726426207936)>, KytosEvent('kytos/topology.links.metadata.added', {'link': Link(Interface('s2-eth3', 3, Switch('00:00:00:00:00:00:00:02')), Inte
rface('s3-eth2', 2, Switch('00:00:00:00:00:00:00:03'))), 'metadata': {'bandwidth': 100, 'ownership': {'red': {}}, 'link_name': 'link-S2-S3'}}, 0)) traceback: Traceback (most recent call
 last):,   File "/home/viniarck/repos/kytos/kytos/core/helpers.py", line 146, in handler_context_apm,     result = handler(*args),   File "/home/viniarck/repos/napps/napps/kytos/pathfin
der/main.py", line 244, in on_links_metadata_changed,     self.graph.update_link_metadata(link),   File "/home/viniarck/repos/napps/napps/kytos/pathfinder/graph.py", line 82, in update_
link_metadata,     self.graph[endpoint_a][endpoint_b][key] = value,   File "/home/viniarck/repos/kytos/.direnv/python-3.9.12/lib/python3.9/site-packages/networkx/classes/coreviews.py",
line 54, in __getitem__,     return self._atlas[key], KeyError: '00:00:00:00:00:00:00:03:2',
kytos $>

In this case, it's a problem where a sequence of threaded events matter, we could use other threading event primitives however, at the moment, it's not guaranteed that an kytos.topology.updated will be sent before kytos/topology.links.metadata.(added|removed), so using a reconciliation since it'll be done in a few milliseconds sounds like a reasonable solution, that way it won't crash and the graph will be updated accordingly shortly after.