search

prometheus-operator / kube-prometheus

Use Prometheus to monitor Kubernetes and applications running on Kubernetes
https://prometheus-operator.dev/
Apache License 2.0
6.73k stars 1.93k forks source link

9093: connect: connection refused #653

Closed JvvYou closed 4 years ago

JvvYou commented 4 years ago

kubernetes version 1.18.6 docker version 19.03.12

`[root@k8s-master-pro-1 kube-prometheus]# kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/alertmanager-main-0 1/2 Running 0 23s pod/alertmanager-main-1 1/2 Running 0 23s pod/alertmanager-main-2 1/2 Running 0 23s pod/grafana-67dfc5f687-g74cj 1/1 Running 0 22s pod/kube-state-metrics-69d4c7c69d-xq58k 3/3 Running 0 21s pod/node-exporter-257c5 2/2 Running 0 21s pod/node-exporter-dtznp 2/2 Running 0 21s pod/node-exporter-p7qhp 2/2 Running 0 21s pod/prometheus-adapter-66b855f564-9l79r 1/1 Running 0 21s pod/prometheus-k8s-0 3/3 Running 1 21s pod/prometheus-k8s-1 3/3 Running 1 21s pod/prometheus-operator-57859b8b59-xd8gm 2/2 Running 0 56s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alertmanager-main ClusterIP 10.111.96.111 9093/TCP 23s service/alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 23s service/grafana ClusterIP 10.106.211.208 3000/TCP 23s service/kube-state-metrics ClusterIP None 8443/TCP,9443/TCP 22s service/node-exporter ClusterIP None 9100/TCP 22s service/prometheus-adapter ClusterIP 10.98.200.226 443/TCP 22s service/prometheus-k8s ClusterIP 10.100.110.143 9090/TCP 22s service/prometheus-operated ClusterIP None 9090/TCP 22s service/prometheus-operator ClusterIP None 8443/TCP 57s

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-exporter 3 3 3 3 3 kubernetes.io/os=linux 22s

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/grafana 1/1 1 1 23s deployment.apps/kube-state-metrics 1/1 1 1 22s deployment.apps/prometheus-adapter 1/1 1 1 22s deployment.apps/prometheus-operator 1/1 1 1 57s

NAME DESIRED CURRENT READY AGE replicaset.apps/grafana-67dfc5f687 1 1 1 23s replicaset.apps/kube-state-metrics-69d4c7c69d 1 1 1 22s replicaset.apps/prometheus-adapter-66b855f564 1 1 1 22s replicaset.apps/prometheus-operator-57859b8b59 1 1 1 57s

NAME READY AGE statefulset.apps/alertmanager-main 0/3 23s statefulset.apps/prometheus-k8s 2/2 22s`

kubectl describe pod/alertmanager-main-0 -n monitoring Warning Unhealthy 2s (x4 over 32s) kubelet, k8s-node-pro-2 Liveness probe failed: Get http://10.244.1.41:9093/-/healthy: dial tcp 10.244.1.41:9093: connect: connection refused Warning Unhealthy 1s (x8 over 36s) kubelet, k8s-node-pro-2 Readiness probe failed: Get http://10.244.1.41:9093/-/ready: dial tcp 10.244.1.41:9093: connect: connection refused

i dont know why 9093 refused,pls help me ,ths

JvvYou commented 4 years ago

[root@k8s-master-pro-1 kube-prometheus]# kubectl logs pod/alertmanager-main-0 alertmanager -n monitoring level=info ts=2020-08-20T15:29:31.257Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=HEAD, revision=4c6c03ebfe21009c546e4d1e9b92c371d67c021d)" level=info ts=2020-08-20T15:29:31.257Z caller=main.go:217 build_context="(go=go1.14.4, user=root@dee35927357f, date=20200617-08:54:02)"

[root@k8s-master-pro-1 kube-prometheus]# kubectl logs pod/alertmanager-main-0 config-reloader -n monitoring 2020/08/20 15:21:10 Watching directory: "/etc/alertmanager/config"

JvvYou commented 4 years ago

i use this

add paused:true

kubectl edit alertmanagers.monitoring.coreos.com

dump

kubectl get statefulset.apps/alertmanager-main -o yaml > dump.yaml

delete the old one

kubectl delete statefulsets.apps alertmanager-main

change statefulset.apps/alertmanager-main to hostNetwork: true

remove both liveness and readiness probe (for the time being)

kubectl create -f dump.yaml

pods run

[root@k8s-master-pro-1 ~]# kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/alertmanager-main-0 2/2 Running 0 4m8s pod/alertmanager-main-1 2/2 Running 0 4m8s

s-urbaniak commented 4 years ago

Which version of kube-prometheus are you using? Additionally, what environment did you deploy kube-prometheus. Your docker version indicates this is some sort of local setup?

xoxodevops commented 4 years ago

I faced the same issue. But it turned out to be that the cluster setup was wrong. I had used calico's single node cluster mechanism whereas, actually i had 3 nodes. Once i corrected my cluster, kube-prometheus worked like a charm !!!!

boxrice007 commented 4 years ago

你有没有做出来

JvvYou commented 4 years ago

I found the problem with my environment, not With Kube-Prometheus

pshyms commented 3 years ago

@weiwei-9527 what do you mean about your env, i face same problem. please help~

veryrich commented 3 years ago

what do you mean about your env, i face same problem. please help~

pshyms commented 3 years ago
  1. use netstat -lntup | grep 9093 to check if port is used by other process
  2. try to set replicas as 1 in yaml
JvvYou commented 3 years ago

what do you mean about your env, i face same problem. please help~

i forgot setup linux kernel params when i install my k8s,so my k8s cluster network has problem

zxl00 commented 3 years ago

me too can you help me

geray-zsg commented 3 years ago

为什么你们都要用英文，我也遇到了同样的问题，但是你们发的我看不懂哈，解决了没

mroldx commented 2 years ago

天坑啊，解决了没兄弟们

Mrccay commented 1 year ago

天坑啊，解决了没有兄弟们,我也遇到这个问题

BoWeiL commented 10 months ago

前面JvvYou的方法是可以的。 只是有些地方要适配自己的环境做修改。

选项 -n 后面接alertmanager-main所在namespace

kubectl -n kubesphere-monitoring-system edit alertmanagers.monitoring.coreos.com

导出statefulset.apps/alertmanager-main的yaml文件

kubectl -n kubesphere-monitoring-system get statefulset.apps/alertmanager-main -o yaml > dump.yaml

删除旧的statefulset.apps/alertmanager-main

kubectl -n kubesphere-monitoring-system delete statefulsets.apps alertmanager-main

修改导出的dump.yaml文件，添加

hostNetwork: true

spec: ... spec: hostNetwork: true #<<添加在这里 affinity: podAntiAffinity:`

重新创建即可

kubectl create -f dump.yaml