search

robusta-dev / kubernetes-chatgpt-bot

A ChatGPT bot for Kubernetes issues.
925 stars 101 forks source link

panic: Unable to create mmap-ed active query log when creating prometheus-robusta-kube-prometheus-st-prometheus-0 pod #11

Closed ashuvaid closed 1 year ago

ashuvaid commented 1 year ago

root@nef:~# kubectl logs prometheus-robusta-kube-prometheus-st-prometheus-0 ts=2023-03-24T11:09:16.291Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.40.5, branch=HEAD, revision=44af4716c86138869aa621737139e6dacf0e2550)" ts=2023-03-24T11:09:16.292Z caller=main.go:561 level=info build_context="(go=go1.19.3, user=root@70f803b28803, date=20221201-12:50:06)" ts=2023-03-24T11:09:16.292Z caller=main.go:562 level=info host_details="(Linux 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 prometheus-robusta-kube-prometheus-st-prometheus-0 (none))" ts=2023-03-24T11:09:16.292Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)" ts=2023-03-24T11:09:16.292Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" ts=2023-03-24T11:09:16.292Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied" panic: Unable to create mmap-ed active query log

goroutine 1 [running]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7fff254b9126, 0xb}, 0x14, {0x3b92be0, 0xc0000988c0}) /app/promql/query_logger.go:121 +0x3cd main.main() /app/cmd/prometheus/main.go:618 +0x6973

ashuvaid commented 1 year ago

Actually a quick remedy was found for the issue I reported. Issue was with the permission of the persistent volume mount point /mnt

root@nef:/mnt/data1# ls -lrt total 4 drwxr-xr-x 2 root root 4096 Mar 24 11:08 prometheus-db

root@nef:/mnt# chmod 777 -R data1/

root@nef:/mnt# ls -lrt total 8 drwxr-xr-x 3 root root 4096 Mar 24 10:18 data drwxrwxrwx 3 root root 4096 Mar 24 11:08 data1

After changing the permissions deleted the pod so that the statefulset will recreate the pod.

NAME READY AGE statefulset.apps/alertmanager-robusta-kube-prometheus-st-alertmanager 1/1 46h statefulset.apps/prometheus-robusta-kube-prometheus-st-prometheus 0/1 46h

root@nef:/mnt# kubectl delete pod prometheus-robusta-kube-prometheus-st-prometheus-0 pod "prometheus-robusta-kube-prometheus-st-prometheus-0" deleted

root@nef:/mnt# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE default alertmanager-robusta-kube-prometheus-st-alertmanager-0 2/2 Running 3 (29m ago) 46h default prometheus-robusta-kube-prometheus-st-prometheus-0 2/2 Running 0 5m6s

Closing the issue