Closed AlexanderZhirov closed 1 year ago
Maybe which parameter forgot to specify? Has anyone ever encountered this? Maybe I'm not being too attentive? I have already checked everything and tested it several times.
I never had this problem before. What model is your Raspberry?
What model is your Raspberry?
Raspberry Pi 4B
I have all the containers raised, the logs are higher. There are no critical errors. Prometheus writes DB, files I see. But why doesn't Grafana output anything? Just empty graphs, as if it is frozen, waiting for a connection. Strange. How can I debug?
My config:
version: "3"
services:
cadvisor:
container_name: monitoring-cadvisor
devices:
- /dev/kmsg:/dev/kmsg
expose:
- 8080
hostname: rpi-cadvisor
image: zcube/cadvisor:latest
ipc: shareable
networks:
- rpimonitor_default
privileged: true
restart: unless-stopped
security_opt:
- label=disable
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
- /etc/machine-id:/etc/machine-id:ro
- /var/lib/dbus/machine-id:/var/lib/dbus/machine-id:ro
grafana:
container_name: monitoring-grafana
user: "0"
environment:
- GF_USERS_ALLOW_SIGN_UP=false
- GF_SECURITY_ADMIN_USER=myuser
- GF_SECURITY_ADMIN_PASSWORD=mypass
- GF_PATHS_CONFIG=/etc/grafana/grafana.ini
- GF_PATHS_DATA=/var/lib/grafana
- GF_PATHS_HOME=/usr/share/grafana
- GF_PATHS_LOGS=/var/log/grafana
- GF_PATHS_PLUGINS=/var/lib/grafana/plugins
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
hostname: rpi-grafana
image: grafana/grafana:latest
networks:
- rpimonitor_default
ports:
- 3000:3000
restart: unless-stopped
volumes:
- /root/docker/monitoring/data/grafana/grafana:/var/lib/grafana
# - /path/to/your/grafana/grafana.ini:/etc/grafana/grafana.ini
# - /path/to/your/grafana/provisioning:/etc/grafana/provisioning
node-exporter:
container_name: monitoring-node-exporter
expose:
- 9100
hostname: rpi-exporter
image: prom/node-exporter:latest
networks:
- rpimonitor_default
restart: unless-stopped
prometheus:
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
container_name: monitoring-prometheus
user: "0"
expose:
- 9090
hostname: rpi-prom
image: prom/prometheus:latest
networks:
- rpimonitor_default
restart: unless-stopped
volumes:
- /root/docker/monitoring/data/prometheus:/prometheus
- /root/docker/monitoring/prometheus:/etc/prometheus/
links:
- cadvisor:cadvisor
- node-exporter:node-exporter
networks:
rpimonitor_default:
external: true
@oijkn Maybe what parameters are incorrect in the docker-compose configuration?
I have the same problem. Raspberry PI 4 + latest Raspbian Bullseye + latest Docker
I'm working on new version, be a little patient :)
I'm working on new version, be a little patient :)
Thanks 🙂
@AlexanderZhirov can you try my last update, check the README please.
I have a recurring Grafana error message with the latest version:: Invalid dashboard UID in annotation request
And the "Install Dashboard from Grafana.com" not working for me, because I have 404 error.
Log:
logger=live.features t=2023-01-17T08:52:51.125002573Z level=error msg="Error getting dashboard" query="{Slug: Id:0 Uid:Ss3q6hSZk OrgId:1 Result:
Ss3q6hSZk UID is the problem?
Sorry for my poor English! :-)
Your error is quite strange, I do not reproduce this problem locally. As you can see the UID is the same in the json file.
Have you tried an installation from scratch ?
Actually I can't download the dash from Grafana, here is the error message: "Too many requests. Please try again later."
Yes, I've tried to restart from scratch several times. I also deleted all generated data.
It seems to work, but the error message keeps popping up. And the Docker Engine info is empty.
First start log:
logger=context userId=1 orgId=1 uname=admin t=2023-01-17T10:34:41.700411937Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=172.23.0.1 time_ms=2 duration=2.205526ms size=0 referer= handler=/api/live/ws
logger=live t=2023-01-17T10:34:41.914232637Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/Ss3q6hSZk address=grafana/dashboard/uid/Ss3q6hSZk
logger=live.features t=2023-01-17T10:34:41.916280388Z level=error msg="Error getting dashboard" query="{Slug: Id:0 Uid:Ss3q6hSZk OrgId:1 Result:<nil>}" error="Dashboard not found"
logger=context userId=1 orgId=1 uname=admin t=2023-01-17T10:34:42.130817856Z level=error msg="Invalid dashboard UID in annotation request" error="Dashboard not found" remote_addr=172.23.0.1 traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-01-17T10:34:42.130971854Z level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=172.23.0.1 time_ms=1 duration=1.75668ms size=70 referer="http://10.0.0.10:3001/?from=now-5m&orgId=1&refresh=5s&to=now" handler=/api/annotations
logger=context userId=1 orgId=1 uname=admin t=2023-01-17T10:34:46.998361626Z level=error msg="Invalid dashboard UID in annotation request" error="Dashboard not found" remote_addr=172.23.0.1 traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-01-17T10:34:46.998526772Z level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=172.23.0.1 time_ms=3 duration=3.361974ms size=70 referer="http://10.0.0.10:3001/?from=now-5m&orgId=1&refresh=5s&to=now" handler=/api/annotations
I'm using port 3001, but I've modified it wherever necessary.
I don't understand what's going on with you... Here are the logs when I start Grafana for the first time :
root@pi:/data/monitoring# docker logs -f monitoring-grafana
logger=settings t=2023-01-17T11:30:50.166159726Z level=info msg="Starting Grafana" version=9.3.2 commit=21c1d14e91 branch=HEAD compiled=2022-12-14T10:40:18Z
logger=settings t=2023-01-17T11:30:50.16863562Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-01-17T11:30:50.168728509Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2023-01-17T11:30:50.168763175Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2023-01-17T11:30:50.169598041Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2023-01-17T11:30:50.16974667Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2023-01-17T11:30:50.169828762Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2023-01-17T11:30:50.169847632Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2023-01-17T11:30:50.16990791Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2023-01-17T11:30:50.169929224Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2023-01-17T11:30:50.169946558Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2023-01-17T11:30:50.170030668Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2023-01-17T11:30:50.170083501Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
logger=settings t=2023-01-17T11:30:50.17010589Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
logger=settings t=2023-01-17T11:30:50.170192704Z level=info msg="Config overridden from Environment variable" var="GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards/rpi-monitoring.json"
logger=settings t=2023-01-17T11:30:50.17022813Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
logger=settings t=2023-01-17T11:30:50.17025963Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-01-17T11:30:50.170278704Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2023-01-17T11:30:50.170294574Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2023-01-17T11:30:50.170309574Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-01-17T11:30:50.170325556Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2023-01-17T11:30:50.170351296Z level=info msg="App mode production"
logger=sqlstore t=2023-01-17T11:30:50.178188791Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2023-01-17T11:30:50.178852047Z level=info msg="Creating SQLite database file" path=/var/lib/grafana/grafana.db
logger=migrator t=2023-01-17T11:30:50.194410389Z level=info msg="Starting DB migrations"
[.......]
logger=provisioning.alerting t=2023-01-17T11:30:55.948537498Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2023-01-17T11:30:55.948566849Z level=info msg="finished to provision alerting"
logger=ngalert.state.manager t=2023-01-17T11:30:55.958906701Z level=info msg="Warming state cache for startup"
logger=grafanaStorageLogger t=2023-01-17T11:30:55.960839246Z level=info msg="storage starting"
logger=http.server t=2023-01-17T11:30:55.964966594Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=ngalert.state.manager t=2023-01-17T11:30:56.01942006Z level=info msg="State cache has been initialized" states=0 duration=60.505062ms
logger=ticker t=2023-01-17T11:30:56.019598744Z level=info msg=starting first_tick=2023-01-17T11:31:00Z
logger=ngalert.multiorg.alertmanager t=2023-01-17T11:30:56.01965117Z level=info msg="starting MultiOrg Alertmanager"
logger=sqlstore.transactions t=2023-01-17T11:30:56.197822209Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked"
logger=sqlstore.transactions t=2023-01-17T11:30:56.388109276Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked"
logger=context t=2023-01-17T11:31:01.938581687Z level=warn msg="failed to look up session from cookie" error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-01-17T11:31:01.93951183Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.12 time_ms=1 duration=1.707102ms size=27 referer= handler=/api/live/ws
logger=context userId=0 orgId=0 uname= t=2023-01-17T11:31:15.951536644Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.12 time_ms=0 duration=846.476µs size=27 referer= handler=/api/live/ws
logger=context userId=0 orgId=0 uname= t=2023-01-17T11:31:30.973058295Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.12 time_ms=1 duration=1.117568ms size=27 referer= handler=/api/live/ws
@oijkn I started from scratch, I redid all my modifications. Now it's perfect! Sorry, I missed the setting somewhere earlier!
@AlexanderZhirov I'm really sorry I deleted your last message by mistake, excuse me.
@oijkn Everything is fine. I will write to you soon about testing when I check it at home.
When I start, I get this:
$ docker-compose up -d node-exporter
Creating network "monitoring_internal" with driver "bridge"
Pulling node-exporter (prom/node-exporter:latest)...
latest: Pulling from prom/node-exporter
15a8001affa7: Pull complete
0ae1023ba014: Pull complete
ca99d81cd0d1: Pull complete
Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c
Status: Downloaded newer image for prom/node-exporter:latest
Creating monitoring-node-exporter ... error
ERROR: for monitoring-node-exporter Cannot start service node-exporter: path / is mounted on / but it is not a shared or slave mount
ERROR: for node-exporter Cannot start service node-exporter: path / is mounted on / but it is not a shared or slave mount
ERROR: Encountered errors while bringing up the project.
Arguments in quotation marks didn't help either. Here is the build data:
raspberry:~/docker/monitoring# uname -a
Linux raspberry 5.15.88-0-rpi4 #1-Alpine SMP PREEMPT Mon Jan 16 17:11:54 UTC 2023 aarch64 Linux
raspberry:~/docker/monitoring# docker --version
Docker version 20.10.21, build baeda1f82a10204ec5708d5fbba130ad76cfee49
raspberry:~/docker/monitoring# docker-compose --version
docker-compose version 1.29.2, build unknown
Portainer says:
I don't understand...
@AlexanderZhirov can you test this : https://ixday.github.io/post/shared_mount/?
@oijkn Yes, everything is working fine now! This method helped to fix the problem. Thank you very much!!! 😌
I did everything according to the instructions. But for some reason, grafana still does not want to display graphs, although all services are working.
monitoring-cadvisor
monitoring-grafana
monitoring-node-exporter
monitoring-prometheus