panic: Unable to create mmap-ed active query log

[vavallee]

Oddly, I have had this chart running without issue on my 3 raspberry pis. I have upgraded to "v1.20.2+k3s1" and suddenly I am seeing this error on the prometheus container.

I've installed the helm chart as per instructions, and in this case used the same version.

level=info ts=2021-02-03T01:07:15.031Z caller=main.go:364 msg="Starting Prometheus" version="(version=2.24.0, branch=HEAD, revision=02e92236a8bad3503ff5eec3e04ac205a3b8e4fe)"
level=info ts=2021-02-03T01:07:15.032Z caller=main.go:369 build_context="(go=go1.15.6, user=root@d9f90f0b1f76, date=20210106-14:46:38)"
level=info ts=2021-02-03T01:07:15.032Z caller=main.go:370 host_details="(Linux 5.4.0-1028-raspi #31-Ubuntu SMP PREEMPT Wed Jan 20 11:30:45 UTC 2021 aarch64 prometheus-prometheus-kube-prometheus-prometheus-0 (none))"
level=info ts=2021-02-03T01:07:15.032Z caller=main.go:371 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2021-02-03T01:07:15.032Z caller=main.go:372 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2021-02-03T01:07:15.032Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0xffffecc5c4bb, 0xb, 0x14, 0x2942c80, 0x4000362ea0, 0x2942c80)
        /app/promql/query_logger.go:117 +0x38c
main.main()
        /app/cmd/prometheus/main.go:400 +0x47e8

[cablespaghetti]

I don't think it's the K8S version that's the problem because I recently rebuilt my setup on that version.

It looks like the key part is file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"'.

Have you upgraded from a previous chart version and kept the same persistent volume? I wonder if a major version of the container has gone from using the root user to a non-root user... If this is the case, do you know the version you when from and to? You should be able to get this with 'helm history Prometheus'.

[cablespaghetti]

Just re-read your post. Sorry I see you didn't change Prometheus versions.

Maybe k3s changed local persistent volume permissions between versions. I'll see what permissions my working setup are using and maybe you can find the directory and chown it.

[vavallee]

Thanks for looking into this. In fact I did destroy all the pvcs before installing, so it appears this happens from scratch. Incidentally, I get the same result without setting a version in the helm deploy.

[vavallee]

I spent an hour trying to get an initContainer: up to fix the permissions, but haven't had any success.

[vavallee]

So can confirm that this is a perms issue on the mount.

I created a separate pod with busybox, mounted the volume under /home/prom and did a chmod -R 777 /home/prom now the service is coming up correctly with the below files created

/home/prom/prometheus-db # ls -l

total 28

drwxr-xr-x    2 1000     2000          4096 Feb 16 19:49 chunks_head

-rw-r--r--    1 1000     2000         20001 Feb 16 19:50 queries.active

drwxr-xr-x    2 1000     2000          4096 Feb 16 19:49 wal

[nemobis]

https://github.com/aws/eks-charts/issues/21

[luckydonald]

https://github.com/prometheus/prometheus/issues/5976