search

hassio-addons / addon-prometheus

Prometheus - Home Assistant Community Add-ons
https://addons.community
MIT License
20 stars 29 forks source link

Prometheus addon crashes during start up #48

Open webwurst opened 2 years ago

webwurst commented 2 years ago

Problem

Prometheus addon crashes during start up.

Logs

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/00-banner.sh
-----------------------------------------------------------
 Add-on: Prometheus
 Cloud native metrics
-----------------------------------------------------------
 Add-on version: fe262a9
 You are running the latest version of this add-on.
 System: Home Assistant OS 8.2  (aarch64 / raspberrypi4-64)
 Home Assistant Core: 2022.6.5
 Home Assistant Supervisor: 2022.05.3
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
cont-init: info: /etc/cont-init.d/00-banner.sh exited 0
cont-init: info: running /etc/cont-init.d/01-log-level.sh
cont-init: info: /etc/cont-init.d/01-log-level.sh exited 0
cont-init: info: running /etc/cont-init.d/prometheus.sh
cont-init: info: /etc/cont-init.d/prometheus.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun prometheus (no readiness notification)
services-up: info: copying legacy longrun prometheus-configgen (no readiness notification)
s6-rc: info: service legacy-services successfully started
[19:48:38] INFO: Starting prometheus...
[19:48:38] INFO: Starting prometheus config generator...
jq: error (at <stdin>:1): null (null) has no keys
ts=2022-06-14T17:48:39.218Z caller=main.go:491 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2022-06-14T17:48:39.219Z caller=main.go:535 level=info msg="Starting Prometheus Server" mode=server version="(version=2.36.1, branch=HEAD, revision=3c0a100dec2950f23e91f7efbf8867750960807d)"
ts=2022-06-14T17:48:39.219Z caller=main.go:540 level=info build_context="(go=go1.18.3, user=root@4827bbf13eda, date=20220609-15:14:08)"
ts=2022-06-14T17:48:39.219Z caller=main.go:541 level=info host_details="(Linux 5.15.32-v8 #1 SMP PREEMPT Thu Jun 9 01:50:59 UTC 2022 aarch64 40817795-prometheus local.hass.io)"
ts=2022-06-14T17:48:39.219Z caller=main.go:542 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-06-14T17:48:39.220Z caller=main.go:543 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-06-14T17:48:39.220Z caller=query_logger.go:90 level=error component=activeQueryTracker msg="Error opening query log file" file=/data/prometheus/queries.active err="open /data/prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7fe8adcc31, 0x10}, 0x14, {0x32e2bb8, 0x4000a144b0})
    /app/promql/query_logger.go:120 +0x2f4
main.main()
    /app/cmd/prometheus/main.go:597 +0x6108
[19:48:39] WARNING: Prometheus crashed, halting add-on
s6-rc: info: service legacy-services: stopping
[19:48:39] INFO: Prometheus config stopped, restarting...
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Additional info

I think the logs contains all the interesting information. I can add more if needed. This also happens after restarting Home Assistant.

webwurst commented 2 years ago

Maybe it's something around these changes?

webwurst commented 2 years ago

Is it possible to install a former release of this addon to verify it's working fine?

42Network commented 2 years ago

Hi @webwurst. I just submitted a PR that allows this EDGE addon to start without issues. It seems to all be working, with both Prometheus and HA metrics showing up. Plus Grafana can see the Prometheus data, too. You can grab the files from my github or just edit the few files by hand.

Now I'll have to decide whether to use Prometheus or InfluxBD for long-term metric storage...

webwurst commented 2 years ago

@42Network is there an easy way for me to test your container image? Like editing some config file with registry/container-name?

webwurst commented 1 year ago

Nice to see an upgrade to Prometheus version 2.40.1 :) Just the permission errors are still persisting and prevent Prometheus to start for me.

amaciuc commented 1 year ago

Any news about this issue? Prometheus add-on is still not working.

PolestarWx commented 1 year ago

Same for me

lupino3 commented 1 year ago

Noting that in the review for #49, @frenck mentioned that prometheus should run as root: https://github.com/hassio-addons/addon-prometheus/pull/49#discussion_r926048521