search

truecharts / public

Community Helm Chart Repository
https://truecharts.org
GNU Affero General Public License v3.0
1.13k stars 617 forks source link

Prometheus node-exporter not starting #9163

Closed gismo2004 closed 1 year ago

gismo2004 commented 1 year ago

App Name

prometheus

SCALE Version

22.12.2

App Version

2.44.0_9.0.10

Application Events

2023-05-25 16:26:37
Error: failed to start container "prometheus-node-exporter": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/dcff70f2-40d5-40a8-a3c7-56d0b2337288/volumes/kubernetes.io~csi/pvc-d3806d55-5b31-4357-87da-b5b857d08540/mount" to rootfs at "/host/proc": mkdir /mnt/Storage1/ix-applications/docker/overlay2/870cc3933c7be939a3b0fb57ab97b7a0c755c323afa1f430d9059e0e13428c05/merged/host/proc: read-only file system: unknown
2023-05-25 16:26:13
Error: failed to start container "prometheus-node-exporter": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/dcff70f2-40d5-40a8-a3c7-56d0b2337288/volumes/kubernetes.io~csi/pvc-d3806d55-5b31-4357-87da-b5b857d08540/mount" to rootfs at "/host/proc": mkdir /mnt/Storage1/ix-applications/docker/overlay2/997922f128203f723a8dc291099e0d7f5e15040c63cdf5dcf6abded2680e449c/merged/host/proc: read-only file system: unknown
2023-05-25 16:26:07
Startup probe failed: HTTP probe failed with statuscode: 503
2023-05-25 16:26:06
Started container config-reloader
2023-05-25 16:26:05
Created container config-reloader
2023-05-25 16:26:04
Container image "quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1" already present on machine
2023-05-25 16:26:04
Created container alertmanager
2023-05-25 16:26:04
Started container alertmanager
2023-05-25 16:26:02
Container image "quay.io/prometheus/alertmanager:v0.24.0" already present on machine
2023-05-25 16:26:02
Back-off restarting failed container
2023-05-25 16:26:01
Error: failed to start container "prometheus-node-exporter": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/dcff70f2-40d5-40a8-a3c7-56d0b2337288/volumes/kubernetes.io~csi/pvc-d3806d55-5b31-4357-87da-b5b857d08540/mount" to rootfs at "/host/proc": mkdir /mnt/Storage1/ix-applications/docker/overlay2/944fcf960e1a50d6c7e26d02fa054fe3f8c3534d42ea7875d159c10723692e9e/merged/host/proc: read-only file system: unknown
2023-05-25 16:26:00
Startup probe failed: Get "http://172.16.0.92:9090/-/ready": dial tcp 172.16.0.92:9090: connect: connection refused
2023-05-25 16:25:58
Created container config-reloader
2023-05-25 16:25:58
Error: failed to start container "prometheus-node-exporter": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/dcff70f2-40d5-40a8-a3c7-56d0b2337288/volumes/kubernetes.io~csi/pvc-ffba085c-f371-4f86-9acc-9afdf10e5bdc/mount" to rootfs at "/host/sys": mkdir /mnt/Storage1/ix-applications/docker/overlay2/e12d638df396bf0ecf63b2614f04dd8de6b0698e2634b2ab0fd3ed51ff3b0c5c/merged/host/sys: read-only file system: unknown
2023-05-25 16:25:58
Started container config-reloader
2023-05-25 16:25:56
Created container prometheus
2023-05-25 16:25:56
Started container prometheus
2023-05-25 16:25:56
Container image "quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1" already present on machine
2023-05-25 16:25:55
Error: failed to start container "prometheus-node-exporter": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/dcff70f2-40d5-40a8-a3c7-56d0b2337288/volumes/kubernetes.io~csi/pvc-d3806d55-5b31-4357-87da-b5b857d08540/mount" to rootfs at "/host/proc": mkdir /mnt/Storage1/ix-applications/docker/overlay2/1ab9bac970268f30b98f9473eab518b48208e555dee572c7055f30c602f3cd70/merged/host/proc: read-only file system: unknown
2023-05-25 16:25:54
Successfully pulled image "quay.io/prometheus/prometheus" in 4.079502905s
2023-05-25 16:25:54
Created container prometheus-node-exporter
2023-05-25 16:25:53
Started container init-config-reloader
2023-05-25 16:25:53
Pulling image "quay.io/prometheus/prometheus"
2023-05-25 16:25:53
Container image "tccr.io/truecharts/node-exporter:v1.5.0@sha256:674e04af703ffb85daf5cbddc64c5fc92e75ba49a5e2b0c0d14a2a8ccace3590" already present on machine
2023-05-25 16:25:52
Created container init-config-reloader
2023-05-25 16:25:51
Add eth0 [172.16.0.93/16] from ix-net
2023-05-25 16:25:49
Container image "quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1" already present on machine
2023-05-25 16:25:41
Unable to attach or mount volumes: unmounted volumes=[host proc sys], unattached volumes=[devshm host proc shared sys tmp varlogs varrun]: timed out waiting for the condition
2023-05-25 16:25:33
create Pod alertmanager-prometheus-alertmanager-0 in StatefulSet alertmanager-prometheus-alertmanager successful
2023-05-25 16:25:33
Successfully assigned ix-prometheus/alertmanager-prometheus-alertmanager-0 to ix-truenas
2023-05-25 16:25:19
Add eth0 [172.16.0.92/16] from ix-net
2023-05-25 16:25:17
Created pod: prometheus-node-exporter-rr6fv
2023-05-25 16:25:17
Successfully assigned ix-prometheus/prometheus-node-exporter-rr6fv to ix-truenas
2023-05-25 16:25:13
Successfully assigned ix-prometheus/prometheus-prometheus-prometheus-0 to ix-truenas
2023-05-25 16:25:13
create Pod prometheus-prometheus-prometheus-0 in StatefulSet prometheus-prometheus-prometheus successful
2023-05-25 16:25:08
Add eth0 [172.16.0.83/16] from ix-net
2023-05-25 16:25:06
Stopping container prometheus-kube-state-metrics
2023-05-25 16:25:06
Started container prometheus-kube-state-metrics
2023-05-25 16:25:04
Created container prometheus-kube-state-metrics
2023-05-25 16:25:02
Container image "tccr.io/truecharts/kube-state-metrics:v2.8.2@sha256:e7b9fbc67f29bb72043238ebaa397d5161f9e3d5cdb16ac888e2ffe152015700" already present on machine
2023-05-25 16:24:55
Updated LoadBalancer with new IPs: [] -> [10.0.0.2]
2023-05-25 16:24:47
Started container prometheus-kube-state-metrics
2023-05-25 16:24:43
Created container prometheus-kube-state-metrics
2023-05-25 16:24:37
Container image "tccr.io/truecharts/kube-state-metrics:v2.8.2@sha256:e7b9fbc67f29bb72043238ebaa397d5161f9e3d5cdb16ac888e2ffe152015700" already present on machine
2023-05-25 16:24:36
There are no available nodes for LoadBalancer
2023-05-25 16:24:20
Updated LoadBalancer with new IPs: [] -> [10.0.0.2]
2023-05-25 16:24:07
Pod sandbox changed, it will be killed and re-created.
2023-05-25 16:23:57
There are no available nodes for LoadBalancer
2023-05-25 16:23:50
Add eth0 [172.16.0.47/16] from ix-net
2023-05-25 16:23:48
MountVolume.MountDevice failed for volume "pvc-ffba085c-f371-4f86-9acc-9afdf10e5bdc" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
2023-05-25 16:23:38
MountVolume.MountDevice failed for volume "pvc-d3806d55-5b31-4357-87da-b5b857d08540" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
2023-05-25 16:23:34
MountVolume.MountDevice failed for volume "pvc-907a0b98-a544-4923-9b10-50ad34ebe2ce" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
2023-05-25 16:23:25
Add eth0 [172.16.0.32/16] from ix-net
2023-05-25 16:23:25
MountVolume.SetUp failed for volume "kube-api-access-zgfmn" : object "ix-prometheus"/"kube-root-ca.crt" not registered
2023-05-25 16:23:24
MountVolume.MountDevice failed for volume "pvc-d47c958a-4bdd-472f-b1f6-c7cafdd80e5b" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
2023-05-25 16:23:23
MountVolume.SetUp failed for volume "config-volume" : object "ix-prometheus"/"alertmanager-prometheus-alertmanager-generated" not registered
2023-05-25 16:23:23
MountVolume.SetUp failed for volume "web-config" : object "ix-prometheus"/"alertmanager-prometheus-alertmanager-web-config" not registered
2023-05-25 16:23:22
MountVolume.SetUp failed for volume "tls-assets" : object "ix-prometheus"/"alertmanager-prometheus-alertmanager-tls-assets-0" not registered
2023-05-25 16:23:18
Cancelling deletion of Pod ix-prometheus/alertmanager-prometheus-alertmanager-0
2023-05-25 16:23:16
Successfully assigned ix-prometheus/prometheus-kube-state-metrics-6bcd7947cd-fvkhw to ix-truenas
2023-05-25 16:23:15
Cancelling deletion of Pod ix-prometheus/prometheus-prometheus-prometheus-0
2023-05-25 16:23:14
Cancelling deletion of Pod ix-prometheus/prometheus-node-exporter-9g2fq
2023-05-25 16:23:13
Cancelling deletion of Pod ix-prometheus/prometheus-kube-state-metrics-6bcd7947cd-w7c8n
2023-05-25 16:23:13
Marking for deletion Pod ix-prometheus/alertmanager-prometheus-alertmanager-0
2023-05-25 16:23:12
Marking for deletion Pod ix-prometheus/prometheus-node-exporter-9g2fq
2023-05-25 16:23:12
Marking for deletion Pod ix-prometheus/prometheus-prometheus-prometheus-0
2023-05-25 16:23:11
Marking for deletion Pod ix-prometheus/prometheus-kube-state-metrics-6bcd7947cd-w7c8n
2023-05-25 16:23:07
Created pod: prometheus-kube-state-metrics-6bcd7947cd-fvkhw
2023-05-25 16:23:07
0/1 nodes are available: 1 node(s) had untolerated taint {ix-svc-start: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
2023-05-25 16:23:06
Deleted pod: prometheus-node-exporter-9g2fq
2023-05-25 16:23:05
network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2023-05-25 16:23:05
network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2023-05-25 16:23:02
MountVolume.SetUp failed for volume "kube-api-access-bnwp2" : object "ix-prometheus"/"kube-root-ca.crt" not registered
2023-05-25 16:22:59
MountVolume.SetUp failed for volume "kube-api-access-7dvwf" : object "ix-prometheus"/"kube-root-ca.crt" not registered
2023-05-25 16:22:58
Ensuring load balancer
2023-05-25 16:22:58
Applied LoadBalancer DaemonSet kube-system/svclb-prometheus-alertmanager-265ec8c8
2023-05-25 16:22:58
There are no available nodes for LoadBalancer
2023-05-25 16:22:58
Ensuring load balancer
2023-05-25 16:22:58
There are no available nodes for LoadBalancer
2023-05-25 16:22:58
Applied LoadBalancer DaemonSet kube-system/svclb-prometheus-fcff6e4c
2023-05-25 16:22:58
MountVolume.MountDevice failed for volume "pvc-dd8081ec-902e-42d4-a077-ebe2ac2525d4" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
2023-05-25 16:22:55
network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2023-05-25 16:22:54
MountVolume.SetUp failed for volume "prometheus-prometheus-prometheus-rulefiles-0" : object "ix-prometheus"/"prometheus-prometheus-prometheus-rulefiles-0" not registered
2023-05-25 16:22:54
MountVolume.SetUp failed for volume "config" : object "ix-prometheus"/"prometheus-prometheus-prometheus" not registered
2023-05-25 16:22:54
MountVolume.SetUp failed for volume "web-config" : object "ix-prometheus"/"prometheus-prometheus-prometheus-web-config" not registered
2023-05-25 16:22:54
MountVolume.SetUp failed for volume "tls-assets" : object "ix-prometheus"/"prometheus-prometheus-prometheus-tls-assets-0" not registered
2023-05-25 16:12:38
Container image "tccr.io/truecharts/node-exporter:v1.5.0@sha256:674e04af703ffb85daf5cbddc64c5fc92e75ba49a5e2b0c0d14a2a8ccace3590" already present on machine
2023-05-25 13:02:45
Back-off restarting failed container

Application Logs

not available for node-exporter since pod is not starting

Application Configuration

image image image image

Nothing changed, default settings

Describe the bug

prometheus is basically working but node-exporter is not starting, thus metrics for it are not available.

To Reproduce

Install prometheus and try to query for "node_exporter_build_info"

Expected Behavior

node-exporter should work if checkbox is set

Screenshots

-

Additional Context

-

I've read and agree with the following

jncanches commented 1 year ago

+1 same error visible in Application events

Error: failed to start container "prometheus-node-exporter": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/dcff70f2-40d5-40a8-a3c7-56d0b2337288/volumes/kubernetes.io~csi/pvc-ffba085c-f371-4f86-9acc-9afdf10e5bdc/mount" to rootfs at "/host/sys": mkdir /mnt/Storage1/ix-applications/docker/overlay2/e12d638df396bf0ecf63b2614f04dd8de6b0698e2634b2ab0fd3ed51ff3b0c5c/merged/host/sys: read-only file system: unknown

PrivatePuffin commented 1 year ago

@jncanches don't make +1 comments unless you want to get banned.