[kitcli] Grafana dashboards are slow to load when port-forwarding

[prateekgogia]

• Either enable host networking mode on Grafana pods and allow the port in security group, or
• Provision a LB for Grafana service Note: For LB we will also need to configure providerID field in the management node.

[RiskyAdventure]

A few thoughts on the problem set and solutions...

For just the port forward issue

• Using managed Grafana SSO or an OIDC auth’d LB.

For the greater problem

• Too many panels on the current dashboards might be using expensive queries such as rate and aggregator functions. Limiting the number of panels by breaking into sub dashboards I have found helpful.

• Reducing the min step to the same as the rate used (4x the scape interval via $__rate_interval) to reduce the number of times Grafana queries the same time series.

• Max data points reduced to effect the min interval (For busy charts that have to many data points when the timeframe is increased i.e. 1hr to 1 day)

• recording rules or “standing queries” that run the aggregator functions so Grafana only needs to run the value of something vs. running in each panel.