In a Charmed Kubernetes environment deployed on Openstack, loadbalancers are not created with health monitoring [0]. While the Provisioning Status for new K8s LBs is Active, the lack of health monitoring causes the Operating Status to be Offline. Therefore, all K8s LBs will trigger the alert for the octavia_loadbalancers check. To deal with this today, we pretty much just add all K8s LBs to the ignore list. However, this is not sustainable, especially when the K8s cluster is heavily used and leads to a lot of noisy alerts if K8s LBs are being constantly created/deleted.
I'm not sure on the best way to address this problem, but here are some ideas of implementation:
implement smarter checking. check to see if health monitors exist before alerting
more extensive ignore logic. perhaps based on regex of names of LBs (as openstack integrator has a set naming scheme
add an option to ignore Operating Status in Loadbalancer checks.
In a Charmed Kubernetes environment deployed on Openstack, loadbalancers are not created with health monitoring [0]. While the Provisioning Status for new K8s LBs is Active, the lack of health monitoring causes the Operating Status to be Offline. Therefore, all K8s LBs will trigger the alert for the octavia_loadbalancers check. To deal with this today, we pretty much just add all K8s LBs to the ignore list. However, this is not sustainable, especially when the K8s cluster is heavily used and leads to a lot of noisy alerts if K8s LBs are being constantly created/deleted.
I'm not sure on the best way to address this problem, but here are some ideas of implementation:
[0] LP#1853668
Imported from Launchpad using lp2gh.
date created: 2021-03-18T14:19:43Z
owner: zzehring
assignee: None
the launchpad url