mohan-nagandlla
commented
3 years ago Instead of kubectl get pod -n monitoring give follow command Kubectl get all -n monitoring
There you can able to see sts alertmanager check whether it is 1/1 or 0/1 if it is 0/1 then describe that sts u will get clarity what was happen exactly
Thank you Mohan nagandlla
On Thu, 15 Apr, 2021, 1:07 pm Ivan Ershov, @.***> wrote:
Describe the bug I'm launching the whole prometheus stack using helmfile and prometheus-kube-stack chart, and can see everything is ok, but alertmanager pod is completely missing:
$ kubectl get pods -n monitoringNAME READY STATUS RESTARTS AGEmonitoring-grafana-667f4cc99b-wkhp6 2/2 Running 0 13mmonitoring-kube-prometheus-operator-f5f75f765-rgbww 1/1 Running 0 13mmonitoring-kube-state-metrics-6c94f8f974-t96bc 1/1 Running 0 13mmonitoring-prometheus-node-exporter-25vtf 1/1 Running 0 13mmonitoring-prometheus-node-exporter-8mdsk 1/1 Running 0 13mmonitoring-prometheus-node-exporter-ccbhh 1/1 Running 0 13mmonitoring-prometheus-node-exporter-ql84j 1/1 Running 0 13mmonitoring-prometheus-node-exporter-qwbxk 1/1 Running 0 13mmonitoring-prometheus-node-exporter-zl2cb 1/1 Running 0 13mprometheus-monitoring-kube-prometheus-prometheus-0 2/2 Running 0 13mprometheus-msteams-76c6df678d-68df9 1/1 Running 0 14m
Version of Helm and Kubernetes:
Helm Version:
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}
Kubernetes Version:
$ kubectl versionClient Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"}Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Which chart: prometheus-kube-stack Which version of the chart: 14.9.0 What happened: Alertmanager pod is missing What you expected to happen: Expected alertmanager to be up and running as all other components
How to reproduce it (as minimally and precisely as possible):
The helm command that you execute and failing/misfunctioning: I'm using helmfile with values from file
helmfile -l group=monitoring apply
Helm values set after installation/upgrade:
USER-SUPPLIED VALUES:additionalPrometheusRulesMap: kafka-lagging: groups: - name: kafka-lagging rules: - expr: sum without (partition) (max(kafka_offset) by (instance, cluster, partition, topic, env)) record: kafka:producer_offset_max - expr: sum without (partition) (min(cg_kafka_offset) by (instance, cluster, partition, topic, env)) record: kafka:consumer_offset_min - expr: sum(rate(cg_kafka_offset[5m])) by (instance, cluster, topic, env) record: kafka:consumer_rate - expr: kafka:producer_offset_max - kafka:consumer_offset_min record: kafka:consumer_lag - expr: kafka:consumer_lag / kafka:consumer_rate record: kafka:consumer_lag_seconds - alert: KafkaConsumerLagSeconds annotations: description: | Kafka consumer on cluster {$labels.cluster} topic {$labels.topic} env {$labels.env} is lagging: Current time lag is { with printf "kafka:consumer_lag_seconds{cluster='%s',topic='%s'}" $labels.cluster $labels.topic | query }{ . | first | value | humanizeDuration }{ end }. Current offset diff is { with printf "kafka:consumer_lag{cluster='%s',topic='%s'}" $labels.cluster $labels.topic | query }{ . | first | value | humanize }{ end }. summary: | Kafka consumer lag is more than 3 minutes or offset difference more than 1k for 5 minutes expr: | kafka:consumer_lag{env="dev"} > 1000 or kafka:consumer_lag_seconds{env="dev"} > 180 for: 5m labels: component: stream-processor severity: warningalertmanager: config: global: resolve_timeout: 5m receivers: - email_configs: - auth_identity: REDACTED auth_password: REDACTED auth_username: REDACTED from: REDACTED smarthost: REDACTED to: REDACTED name: k8s-admin - name: msteams webhook_configs: - send_resolved: true url: http://prometheus-msteams:2000/alertmanager route: - receiver: k8s-admin routes: [] - receiver: msteams routes: [] enabled: truedefaultRules: rules: kubeScheduler: falsegrafana: adminPassword: REDACTED enabled: true ingress: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/ssl-redirect: "true" enabled: true hosts: - REDACTED paths: - /kubeControllerManager: enabled: falseprometheus: prometheusSpec: podMonitorSelectorNilUsesHelmValues: false probeSelectorNilUsesHelmValues: false ruleSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false
Anything else we need to know:
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/prometheus-community/helm-charts/issues/851, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMR7ZPTIAYC3MWYEW4GMRIDTI2JTBANCNFSM4264W2BQ .
iershovnsk
qmiinh
OlivierMary
eduartua
stale[bot]
mertyakan
project-administrator
K890
EminBA
UntouchedWagons
Describe the bug I'm launching the whole prometheus stack using helmfile and prometheus-kube-stack chart, and can see everything is ok, but alertmanager pod is completely missing:
Version of Helm and Kubernetes:
Helm Version:
Kubernetes Version:
Which chart: prometheus-kube-stack Which version of the chart: 14.9.0 What happened: Alertmanager pod is missing What you expected to happen: Expected alertmanager to be up and running as all other components
How to reproduce it (as minimally and precisely as possible):
The helm command that you execute and failing/misfunctioning: I'm using helmfile with values from file
Helm values set after installation/upgrade:
Anything else we need to know: