search

kytos-ng / kytos

Kytos SDN Platform. Kytos is designed to be easy to install, use, develop and share Network Apps (NApps).
https://kytos-ng.github.io/
MIT License
2 stars 7 forks source link

perf: topology with 100+ evcs usually spike CPU up to 100% during network convergence #478

Open viniarck opened 1 month ago

viniarck commented 1 month ago

kytosd is spiking up to 100% CPU usage (measured on a i7-12700H 2.3 GHz)

Locally, I'm using a topology with 100 EVCs a performing one link flap on both current_path and in the failover_path, during flow mod removals and installations it's spiking up to 100% during that period, this is supposed to be mostly IO-bound, this needs investigation to profile and see if we can spot the main sources cause that, maybe msg serialization/deserialization might be struggling, still needs to be profiled:

cpu

cpu1

On APM these were the top ranked traced handlers running at the time:

20240524_121259

Ktmi commented 3 weeks ago

So it took some effort to recreate, but here's what I got. First I created a few hundred evcs on Kytos, then I deleted all the created flows on open vswitch. I then restarted kytos, with consistency check disabled on flow manager, to allow for resending of all stored flows to execute. This then causes a brief period where the kytos process is at 100% cpu utilization.