ravindraprasad85
commented
1 year ago I fixed this issue by changing the istio rules. Below link helped me to fix my issue https://github.com/istio/tools/blob/c034c33637f6441cfc3c921b184b5a9ba1301d68/perf/stability/alertmanager/prometheusrule.yaml
Describe the bug a clear and concise description of what the bug is.
I am trying to add my custom Prometheusrules for istio but those are not visible in either "Alerts" section or "Status/Rules" section. However, rest of all alert rules showing appropriately in portal. istio PrometheusRules are also getting created but not listing in below URLs
https://internal-prometheus/rules https://internal-prometheus/alerts?search=
To show here that istio prometheusrules created
kubectl get PrometheusRules -n monitoring
NAME AGE kube-prometheus-stack-alertmanager.rules 14d kube-prometheus-stack-ccid-authn.rules 14d kube-prometheus-stack-ccid-container.rules 14d kube-prometheus-stack-ccid-etcd.rules 14d kube-prometheus-stack-ccid-fluentbit.rules 14d kube-prometheus-stack-ccid-istio.rules 39m kube-prometheus-stack-ccid-nginx.rules 14d kube-prometheus-stack-ccid-portal.rules 14d kube-prometheus-stack-ccid-promo.rules 14d kube-prometheus-stack-ccid-verify.rules 14d kube-prometheus-stack-config-reloaders 14d kube-prometheus-stack-etcd 14d kube-prometheus-stack-general.rules 14d kube-prometheus-stack-k8s.rules 14d kube-prometheus-stack-kube-apiserver-availability.rules 14d kube-prometheus-stack-kube-apiserver-burnrate.rules 14d kube-prometheus-stack-kube-apiserver-histogram.rules 14d kube-prometheus-stack-kube-apiserver-slos 14d kube-prometheus-stack-kube-prometheus-general.rules 14d kube-prometheus-stack-kube-prometheus-node-recording.rules 14d kube-prometheus-stack-kube-state-metrics 14d kube-prometheus-stack-kubelet.rules 14d kube-prometheus-stack-kubernetes-apps 14d kube-prometheus-stack-kubernetes-resources 14d kube-prometheus-stack-kubernetes-storage 14d kube-prometheus-stack-kubernetes-system 14d kube-prometheus-stack-kubernetes-system-apiserver 14d kube-prometheus-stack-kubernetes-system-kube-proxy 14d kube-prometheus-stack-kubernetes-system-kubelet 14d kube-prometheus-stack-node-exporter 14d kube-prometheus-stack-node-exporter.rules 14d kube-prometheus-stack-node-network 14d kube-prometheus-stack-node.rules 14d kube-prometheus-stack-prometheus 14d kube-prometheus-stack-prometheus-operator 14d`
What's your helm version?
version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.18.1"}
What's your kubectl version?
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-fb459a0", GitCommit:"b07006b2e59857b13fe5057a956e86225f0e82b7", GitTreeState:"clean", BuildDate:"2022-10-24T20:32:54Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1
Which chart?
kube-prometheus-stack
What's the chart version?
version: 39.4.0
What happened?
Istio Prometheus alert Rules are not getting created for alerting istio.
What you expected to happen?
istio alerting rules should show on prometheus portal
How to reproduce it?
defaultRules: create: true rules: ccidistio: false
Enter the changed values of values.yaml?
defaultRules: create: true rules: ccidistio: true
Enter the command that you execute and failing/misfunctioning.
helm upgrade kube-prometheus-stack ../kube-prometheus-stack -n monitoring --debug
Anything else we need to know?
`{{- / Generated from 'ccid-istio.rules' group from https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubernetesControlPlane-prometheusRule.yaml Do not change in-place! In order to change this file first read following link: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/hack / -}} {{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }} {{- if and (semverCompare ">=1.14.0-0" $kubeTargetVersion) (semverCompare "<9.9.9-9" $kubeTargetVersion) .Values.defaultRules.create .Values.defaultRules.rules.ccidistio }} {{- $namespace := printf "%s" (include "kube-prometheus-stack.namespace" .) }} apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: {{ printf "%s-%s" (include "kube-prometheus-stack.fullname" .) "ccid-istio.rules" | trunc 63 | trimSuffix "-" }} namespace: {{ template "kube-prometheus-stack.namespace" . }} labels: app: {{ template "kube-prometheus-stack.name" . }} {{ include "kube-prometheus-stack.labels" . | indent 4 }} {{- if .Values.defaultRules.labels }} {{ toYaml .Values.defaultRules.labels | indent 4 }} {{- end }} {{- if .Values.defaultRules.annotations }} annotations: {{ toYaml .Values.defaultRules.annotations | indent 4 }} {{- end }} spec: groups:
{{}}$labels.instance{{}}}} description: "Gateway pods have dropped. Inbound traffic will likely be affected.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.pod{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Number of Istio Pilot push errors is too high (> 5%). Envoy sidecars might have outdated configuration.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Number of Mixer dispatches to Prometheus is too low. Istio metrics might not be being exported properly.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Global request rate in the service mesh is unusually high.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Global request rate in the service mesh is unusually low.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "High percentage of HTTP 5xx responses in Istio (> 5%).\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "High percentage of HTTP 5xx responses in Istio (> 5%).\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Istio average requests execution is longer than 100ms.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Istio 1% slowest requests are longer than 1s.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: severity: warning app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }}{{}}$labels.instance{{}}) description: "Istio pilot duplicate entry error.\n VALUE = {{{{}}$labels.pod{{}}}}\n LABELS = {{{{}}$labels.instance{{}}}}" labels: action_type: ticket severity: critical app: ccid-istio {{- if .Values.defaultRules.additionalRuleLabels }} {{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }} {{- end }} {{- end }} `