Our 2 core and 4GB of RAM prod machine was getting busy handling the Crawler + DB in a single machine.
The CPU was constantly at 100%, affecting the overall performance of the crawler and the query time to the DB.
It would be nice to profile the CPU usage and make sure that the workload comes from the peering workers and the low peering latency to ~22k peers every ~2mins and not from a wrongly handled loop that eats CPU
Description
Our 2 core and 4GB of RAM prod machine was getting busy handling the Crawler + DB in a single machine. The CPU was constantly at 100%, affecting the overall performance of the crawler and the query time to the DB.
It would be nice to profile the CPU usage and make sure that the workload comes from the
peering workers
and the low peering latency to ~22k peers every ~2mins and not from a wrongly handledloop
that eats CPU