search

kytos-ng / pathfinder

Kytos main path finder Network Application (NApp)
https://kytos-ng.github.io/api/pathfinder.html
MIT License
0 stars 7 forks source link

RuntimeError: dictionary changed size during iteration #24

Closed italovalcy closed 2 years ago

italovalcy commented 2 years ago

Error message:

2022-08-07 18:48:04,396 - ERROR [kytos.core.helpers] [helpers.py:151:handler_context_apm] (thread_pool_app_34) listen_to handler: <function Main.on_topology_updated at 0x7fc84435b4c0>, args: (<Main(pathfinder, stopped 140497104402176)>, KytosEvent('kytos/topology.updated', {'topology': <napps.kytos.topology.models.Topology object at 0x7fc7765378e0>}, 0)) traceback: Traceback (most recent call last):,   File "/usr/local/lib/python3.9/dist-packages/kytos/core/helpers.py", line 146, in handler_context_apm,     result = handler(*args),   File "//var/lib/kytos/napps/kytos/pathfinder/main.py", line 200, in on_topology_updated,     self.update_topology(event),   File "//var/lib/kytos/napps/kytos/pathfinder/main.py", line 209, in update_topology,     self.graph.update_topology(topology),   File "//var/lib/kytos/napps/../napps/kytos/pathfinder/graph.py", line 48, in update_topology,     self.update_links(topology.links),   File "//var/lib/kytos/napps/../napps/kytos/pathfinder/graph.py", line 70, in update_links,     for link in links.values():, RuntimeError: dictionary changed size during iteration,

I'm not 100% about how to reproduce this error, but it just happened when I was doing performance measurements on Kytos. Environment: using docker container with amlight/kytos:latest, Mininet with AmLight Topo, creating 100 EVCs and simulating link failures (after start the emulation, create evcs, setup everything, wait a few seconds and issue a link down - e.g. tmux send-keys -t mn "sh ip link set down SoL2-eth5" ENTER).

viniarck commented 2 years ago

Great, catch @italovalcy. This is related to shared memory and dict mutability. Since this event kytos.topology.updated on topology is mostly for broadcasting the entire topology updated, I believe we should send a new dict copy instead, it won't make it immutable, but since it's sending a copy, if new links or switches are added then it wouldn't result in this issue.

We probably need to document best practices about this since Python allows the developer to shoot themselves in the foot here. A general guideline could be, if you're sending a collection send a copy, if the collection can be too large, then only sends the ids, and then the caller query/get the ids accordingly.