search

sassoftware / viya4-monitoring-kubernetes

Provides simple scripts and customization options to deploy monitoring, alerts, and log aggregation for Viya 4 running on Kubernetes
Apache License 2.0
54 stars 32 forks source link

KubeHpaMaxedOut patching breaks any alterations via user-values file. #575

Closed tynsh closed 10 months ago

tynsh commented 10 months ago

Currently the default set of alerts is deployed using helm to the cluster and then patched inside the cluster to fix an issue with the Kube HPA alert using kubectl patch. This patch specifically references the 14th element in the resulting PrometheusRule manifest. This makes it impossible to modify the list of deployed alerts using parameters to the kube-prometheus-stack via the user-values file.

https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/215be0ce522b354175fea58791180885ee335ad6/monitoring/kube-hpa-alert-patch.json#L4

A better approach is disabling the deployment of the KubeHpaMaxedOut alert via the defaultRules.disabled variable and adding a modified version of it using the appropriate option.

Using this approach is preferred as it also prevents deploying incorrect rules to the cluster intermittently.

gsmith-sas commented 10 months ago

@tynsh Thank you for reporting this issue. Your suggested solution does appear to be a better way of achieving our goals without preventing users from customizing their deployment. I have added an item to our project backlog to implement and validate your suggested fix. I cannot say exactly when it will be incorporated into the project but your detailed suggestion for a fix will make it easier for us to tackle. We will update this issue when the work is completed. Thank you again.