Both the functions collect_pv_capacity_metrics() & collect_pv_utilization_metrics() first clear the respective textfiles holding the metrics info before recreating them with latest updated info. However, the procedure to derive these metrics involves kubectl commands and/or system calls with some output parsing. Under cases where these take more than a few seconds & if the prometheus scrape interval is such that it coincides with this state of the textfiles (empty), then the dashboards are prone to show a few seconds worth of "no-metrics". This manifests itself like shown below:
This should be fixed. The changes may occur in two phases:
The existing bash script will be updated to replace metrics over clearing files
Both the functions
collect_pv_capacity_metrics()&collect_pv_utilization_metrics()first clear the respective textfiles holding the metrics info before recreating them with latest updated info. However, the procedure to derive these metrics involveskubectlcommands and/or system calls with some output parsing. Under cases where these take more than a few seconds & if the prometheus scrape interval is such that it coincides with this state of the textfiles (empty), then the dashboards are prone to show a few seconds worth of "no-metrics". This manifests itself like shown below:This should be fixed. The changes may occur in two phases: