search

STORDIS / monsoon

SONiC monitoring system supplies SONiC telemetry data.
Apache License 2.0
35 stars 10 forks source link

Multithreaded metric collection. #27

Closed kamalkrbh closed 2 years ago

kamalkrbh commented 2 years ago

On slow systems scrap gets timed out and scrap_timeout of prometheus need to be configured to be big enough to fill all the metrics. A multithreaded metric will be more performant on exporter side.

kamalkrbh commented 2 years ago

As per analysis, export_temp_info() function in exporter.py is consuming significant time, without this metric exported all other metrics are exported in ~1 sec. But with temp_info overall time significantly increases to ~17 secs. Introducing multi threaded implementation doesn't seems to solve the problem. Need to further analyse the implementation of export_temp_info().

kamalkrbh commented 2 years ago

Corrected placement of export_hwmon_temp_info, it was misplaced causing nested loop execution and costing execution time. Also introduced threadpool of 10 threads for metric collection. Metric collection takes ~2 secs now.