octavia loadbalancer check is overly sensitive; k8s LBs will always trigger alert

[sudeephb]

In a Charmed Kubernetes environment deployed on Openstack, loadbalancers are not created with health monitoring [0]. While the Provisioning Status for new K8s LBs is Active, the lack of health monitoring causes the Operating Status to be Offline. Therefore, all K8s LBs will trigger the alert for the octavia_loadbalancers check. To deal with this today, we pretty much just add all K8s LBs to the ignore list. However, this is not sustainable, especially when the K8s cluster is heavily used and leads to a lot of noisy alerts if K8s LBs are being constantly created/deleted.

I'm not sure on the best way to address this problem, but here are some ideas of implementation:

• implement smarter checking. check to see if health monitors exist before alerting
• more extensive ignore logic. perhaps based on regex of names of LBs (as openstack integrator has a set naming scheme
• add an option to ignore Operating Status in Loadbalancer checks.

[0] LP#1853668

---

Imported from Launchpad using lp2gh.

• date created: 2021-03-18T14:19:43Z

• owner: zzehring

• assignee: None

• the launchpad url

[sudeephb]

(by peppepetra) Also charm alerts for all lbs with operating status != "ONLINE" while "DRAINING", "NO_MONITOR" should be considered ok as well