search

prometheus-operator / kube-prometheus

Use Prometheus to monitor Kubernetes and applications running on Kubernetes
https://prometheus-operator.dev/
Apache License 2.0
6.62k stars 1.91k forks source link

prometheus-k8s-0 keep Terminating ？ #2444

Open trustnote-wang opened 2 months ago

trustnote-wang commented 2 months ago

k8s version: 1.16.15

kube-prometheus version : release-0.4

This is my first time deploying kube-prometheus，but it not work，pod Prometheus-k8s-0 keeps restarting and terminating after deployment

NAME READY STATUS RESTARTS AGE alertmanager-main-0 0/2 Terminating 0 4s alertmanager-main-1 2/2 Terminating 0 14s grafana-58dc7468d7-26xb2 1/1 Running 0 84m kube-state-metrics-765c7c7f95-s9dgw 3/3 Running 0 84m node-exporter-78jzl 2/2 Running 0 84m node-exporter-9nfd8 0/2 Pending 0 84m node-exporter-b4bv4 2/2 Running 0 84m node-exporter-bjf6x 2/2 Running 0 84m node-exporter-c289g 2/2 Running 0 84m node-exporter-d6xqh 2/2 Running 0 84m node-exporter-f5wls 2/2 Running 0 84m node-exporter-g4qp4 2/2 Running 0 84m node-exporter-j92zw 2/2 Running 0 84m node-exporter-jjt78 2/2 Running 0 84m node-exporter-lpmmg 0/2 Pending 0 84m node-exporter-nmv9t 2/2 Running 0 84m node-exporter-vtzrc 2/2 Running 0 84m node-exporter-wtctm 2/2 Running 0 84m node-exporter-xcbsl 2/2 Running 0 84m node-exporter-xkjdj 2/2 Running 0 84m node-exporter-xsh62 2/2 Running 0 84m prometheus-adapter-5cd5798d96-jx6qz 1/1 Running 0 84m prometheus-k8s-0 0/3 ContainerCreating 0 2s prometheus-k8s-1 0/3 ContainerCreating 0 1s prometheus-operator-5f75d76f9f-g72s4 1/1 Running 0 87m

error message：

Normal Scheduled default-scheduler Successfully assigned monitoring/prometheus-k8s-0 to qde02.test.com Normal Pulled 5s kubelet, qde02.test.com Container image "quay.io/prometheus/prometheus:v2.11.0" already present on machine Warning Failed 5s kubelet, qde02.test.com Error: cannot find volume "config-out" to mount into container "prometheus" Normal Pulled 5s kubelet, qde02.test.com Container image "rancher/coreos-prometheus-config-reloader:v0.38.1" already present on machine Warning Failed 5s kubelet, qde02.test.com Error: cannot find volume "config" to mount into container "prometheus-config-reloader" Normal Pulled 5s kubelet, qde02.test.com Container image "rancher/coreos-configmap-reload:v0.0.1" already present on machine Warning Failed 5s kubelet, qde02.test.com Error: cannot find volume "prometheus-k8s-rulefiles-0" to mount into container "rules-configmap-reloader"

alertmanager error message:

Normal Pulled 3s kubelet, qde02.test.com Container image "quay.io/prometheus/alertmanager:v0.18.0" already present on machine Normal Created 3s kubelet, qde02.test.com Created container alertmanager Warning Failed 3s kubelet, qde02.test.com Error: failed to start container "alertmanager": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/var/lib/kubelet/pods/a0b7a19d-338f-4140-9923-68dcc15bf1ca/volumes/kubernetes.io~empty-dir/alertmanager-main-db" to rootfs at "/alertmanager" caused: stat /var/lib/kubelet/pods/a0b7a19d-338f-4140-9923-68dcc15bf1ca/volumes/kubernetes.io~empty-dir/alertmanager-main-db: no such file or directory: unknown Normal Pulled 3s kubelet, qde02.test.com Container image "rancher/coreos-configmap-reload:v0.0.1" already present on machine Warning Failed 3s kubelet, qde02.test.com Error: cannot find volume "config-volume" to mount into container "config-reloader"

csyyy106 commented 2 months ago

image

Add this to fix it

trustnote-wang commented 2 months ago

image

Add this to fix it

HI ，I tried it, but it didn't work