PODS in pending state as persistent volume is missing

[ashuvaid]

Hi

Below pods are in pending state as PersistentVolumes have not got configured after helm install and helm upgrade of robusta. I have followed the steps mentioned in the readme but this is the state where I have got stuck. Please let me know the workaround.

root@k8sautochatgpt:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default alertmanager-robusta-kube-prometheus-st-alertmanager-0 0/2 Pending 0 22m default prometheus-robusta-kube-prometheus-st-prometheus-0 0/2 Pending 0 22m default robusta-forwarder-bb4fdfb88-vwkxs 1/1 Running 0 23m default robusta-grafana-f6c6b9d4d-4qhxj 3/3 Running 0 23m default robusta-kube-prometheus-st-operator-56b86ffb9f-24bc7 1/1 Running 0 23m default robusta-kube-state-metrics-578cc467c5-xdthx 1/1 Running 0 23m default robusta-prometheus-node-exporter-v4vdt 1/1 Running 0 23m default robusta-runner-8695c9d6f9-2tgcx 1/1 Running 0 23m kube-flannel kube-flannel-ds-jmmsz 1/1 Running 0 84m kube-system coredns-787d4945fb-cth7p 1/1 Running 0 84m kube-system coredns-787d4945fb-rfz2t 1/1 Running 0 84m kube-system etcd-k8sautochatgpt 1/1 Running 0 84m kube-system kube-apiserver-k8sautochatgpt 1/1 Running 0 84m kube-system kube-controller-manager-k8sautochatgpt 1/1 Running 0 84m kube-system kube-proxy-gqvh4 1/1 Running 0 84m kube-system kube-scheduler-k8sautochatgpt 1/1 Running 0 84m

root@k8sautochatgpt:~# kubectl describe pod prometheus-robusta-kube-prometheus-st-prometheus-0 Events: Type Reason Age From Message

---

Warning FailedScheduling 2m22s (x5 over 22m) default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..

root@k8sautochatgpt:~# kubectl describe pod alertmanager-robusta-kube-prometheus-st-alertmanager-0 Events: Type Reason Age From Message

---

Warning FailedScheduling 2m7s (x6 over 23m) default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. root@k8sautochatgpt:~# kubectl get pv,pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/alertmanager-robusta-kube-prometheus-st-alertmanager-db-alertmanager-robusta-kube-prometheus-st-alertmanager-0 Pending 23m persistentvolumeclaim/prometheus-robusta-kube-prometheus-st-prometheus-db-prometheus-robusta-kube-prometheus-st-prometheus-0 Pending 23m root@k8sautochatgpt:~#

[Avi-Robusta]

Hi @ashuvaid can you run kubectl describe on the persistent volume? What provider are you using for kubernetes?

[ashuvaid]

Hi @ashuvaid can you run kubectl describe on the persistent volume? What provider are you using for kubernetes?

Hi @Avi-Robusta - Thanks for your response. There are no PV's created in the setup after helm install of robusta. Howvever PVC are created.

What provider are you using for kubernetes? --> I have used kubeadm to deploy my kubernetes cluster.

[ashuvaid]

Hi @ashuvaid can you run kubectl describe on the persistent volume? What provider are you using for kubernetes?

Hi @Avi-Robusta - Thanks for your response. There are no PV's created in the setup after helm install of robusta. Howvever PVC are created.

What provider are you using for kubernetes? --> I have used kubeadm to deploy my kubernetes cluster.

Issue resolved after creating below persistent volumes

apiVersion: v1 kind: PersistentVolume metadata: name: alertmanager-robusta-kube-prometheus-st-alertmanager-db labels: alertmanager: robusta-kube-prometheus-st-alertmanager spec:

storageClassName: manual

capacity: storage: 10Gi accessModes:

• ReadWriteOnce hostPath: path: "/mnt/data"

apiVersion: v1 kind: PersistentVolume metadata: name: prometheus-robusta-kube-prometheus-st-prometheus-db labels: prometheus: robusta-kube-prometheus-st-prometheus spec:

storageClassName: manual

capacity: storage: 100Gi accessModes:

• ReadWriteOnce hostPath: path: "/mnt/data1"

root@nef:~# kubectl get pods NAME READY STATUS RESTARTS AGE alertmanager-robusta-kube-prometheus-st-alertmanager-0 2/2 Running 3 (16m ago) 46h prometheus-robusta-kube-prometheus-st-prometheus-0 1/2 CrashLoopBackOff 5 (37s ago) 46h

However getting below error for prometheus-robusta-kube-prometheus-st-prometheus-0 pod

root@nef:~# kubectl logs prometheus-robusta-kube-prometheus-st-prometheus-0 ts=2023-03-24T11:09:16.291Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.40.5, branch=HEAD, revision=44af4716c86138869aa621737139e6dacf0e2550)" ts=2023-03-24T11:09:16.292Z caller=main.go:561 level=info build_context="(go=go1.19.3, user=root@70f803b28803, date=20221201-12:50:06)" ts=2023-03-24T11:09:16.292Z caller=main.go:562 level=info host_details="(Linux 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 prometheus-robusta-kube-prometheus-st-prometheus-0 (none))" ts=2023-03-24T11:09:16.292Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)" ts=2023-03-24T11:09:16.292Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" ts=2023-03-24T11:09:16.292Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied" panic: Unable to create mmap-ed active query log

goroutine 1 [running]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7fff254b9126, 0xb}, 0x14, {0x3b92be0, 0xc0000988c0}) /app/promql/query_logger.go:121 +0x3cd main.main() /app/cmd/prometheus/main.go:618 +0x6973

[ashuvaid]

tracking the above error for pod prometheus-robusta-kube-prometheus-st-prometheus-0 as a new issue