search

oijkn / Docker-Raspberry-PI-Monitoring

A docker-compose stack solution for monitoring host and containers with Prometheus, Grafana, cAdvisor and NodeExporter.
MIT License
410 stars 61 forks source link

Network graph dont seems to work for me #13

Closed Akikazuu closed 1 year ago

Akikazuu commented 2 years ago

Hi, i run the stack prometheus, node-exporter, cadvisor, grafana on a raspberry pi 4 8Go with a 500Gb SSD on Ubuntu 21.10

Im tryed to download huge file via wget on the rpi but Network stats dont seems to moove (blue line is a reboot) : image

I use portainer to deploy the stack.

Here the docker-compose : 

version: "2.4"
services:
  cadvisor:
    container_name: monitoring-cadvisor
    devices:
      - /dev/kmsg:/dev/kmsg
    expose:
      - 8080
    hostname: rpi-cadvisor
    image: zcube/cadvisor:latest
    ipc: shareable
    networks:
      - rpimonitor_default
    privileged: true
    restart: unless-stopped
    security_opt:
      - label=disable
    volumes:
      - /:/rootfs:ro
      - /var/run/:/var/run/:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /sys/fs/cgroup/:/cgroup 

  grafana:
    container_name: monitoring-grafana
    image: grafana/grafana:latest
    environment:
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_PATHS_CONFIG=/etc/grafana/grafana.ini
      - GF_PATHS_DATA=/var/lib/grafana
      - GF_PATHS_HOME=/usr/share/grafana
      - GF_PATHS_LOGS=/var/log/grafana
      - GF_PATHS_PLUGINS=/var/lib/grafana/plugins
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
    hostname: rpi-grafana
    networks:
      - rpimonitor_default
    ports:
      - 3000:3000
    restart: unless-stopped
    volumes:
      - /portainer/Files/AppData/Config/grafana/data:/var/lib/grafana
      - /portainer/Files/AppData/Config/grafana/grafana.ini:/etc/grafana/grafana.ini
      - /portainer/Files/AppData/Config/grafana/provisioning:/etc/grafana/provisioning

  node-exporter:
    container_name: monitoring-node-exporter
    expose:
      - 9100
    hostname: rpi-exporter
    image: prom/node-exporter:latest
    networks:
      - rpimonitor_default
    restart: unless-stopped

  prometheus:
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-15d}'
    container_name: monitoring-prometheus
    expose:
      - 9090
    hostname: rpi-prom
    image: prom/prometheus:latest
    networks:
      - rpimonitor_default
    restart: unless-stopped
    user: root
    volumes:
      - /portainer/Files/AppData/Config/prometheus/data:/prometheus
      - /portainer/Files/AppData/Config/prometheus/config:/etc/prometheus/
    links:
      - cadvisor:cadvisor
      - node-exporter:node-exporter

networks:
  rpimonitor_default:
    external: true

Here the node-exporter logs : 

ts=2022-01-26T07:35:39.489Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"
ts=2022-01-26T07:35:39.489Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:10:22)"
ts=2022-01-26T07:35:39.493Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)
ts=2022-01-26T07:35:39.494Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=arp
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=bcache
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=bonding
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=btrfs
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=conntrack
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=cpu
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=cpufreq
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=diskstats
ts=2022-01-26T07:35:39.494Z caller=node_exporter.go:115 level=info collector=dmi
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=edac
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=entropy
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=fibrechannel
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=filefd
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=filesystem
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=hwmon
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=infiniband
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=ipvs
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=loadavg
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=mdadm
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=meminfo
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=netclass
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=netdev
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=netstat
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=nfs
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=nfsd
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=nvme
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=os
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=powersupplyclass
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=pressure
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=rapl
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=schedstat
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=sockstat
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=softnet
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=stat
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=tapestats
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=tcpstat
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=textfile
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=thermal_zone
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=time
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=timex
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=udp_queues
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=uname
ts=2022-01-26T07:35:39.495Z caller=node_exporter.go:115 level=info collector=vmstat
ts=2022-01-26T07:35:39.496Z caller=node_exporter.go:115 level=info collector=xfs
ts=2022-01-26T07:35:39.496Z caller=node_exporter.go:115 level=info collector=zfs
ts=2022-01-26T07:35:39.497Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100
ts=2022-01-26T07:35:39.498Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false

Prometheus logs : 

ts=2022-01-26T07:55:39.151Z caller=main.go:515 level=info msg="Starting Prometheus" version="(version=2.32.1, branch=HEAD, revision=41f1a8125e664985dd30674e5bdf6b683eff5d32)"
ts=2022-01-26T07:55:39.151Z caller=main.go:520 level=info build_context="(go=go1.17.5, user=root@a2a62523c036, date=20211217-21:54:28)"
ts=2022-01-26T07:55:39.151Z caller=main.go:521 level=info host_details="(Linux 5.13.0-1015-raspi #17-Ubuntu SMP PREEMPT Thu Jan 13 01:27:28 UTC 2022 aarch64 rpi-prom (none))"
ts=2022-01-26T07:55:39.151Z caller=main.go:522 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-01-26T07:55:39.151Z caller=main.go:523 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-01-26T07:55:39.160Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-01-26T07:55:39.162Z caller=main.go:924 level=info msg="Starting TSDB ..."
ts=2022-01-26T07:55:39.164Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1643122413941 maxt=1643133600000 ulid=01FT9QFBY2N6AM360T0E2Y4G8Y
ts=2022-01-26T07:55:39.164Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1643155200124 maxt=1643162400000 ulid=01FTA56SGX7CRCX07H5D6KQKC5
ts=2022-01-26T07:55:39.164Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1643133600000 maxt=1643155200000 ulid=01FTA56TQ6QDQPRRFY8R89AD3N
ts=2022-01-26T07:55:39.164Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1643162400023 maxt=1643169600000 ulid=01FTAC2GRT60HZ0S3B3QFB3AT6
ts=2022-01-26T07:55:39.165Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1643169600034 maxt=1643176800000 ulid=01FTAJY80W5FDKZ5M5YX1XBS26
ts=2022-01-26T07:55:39.167Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-01-26T07:55:39.193Z caller=head.go:488 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-01-26T07:55:39.215Z caller=head.go:513 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 13233"
ts=2022-01-26T07:55:39.215Z caller=head.go:668 level=info component=tsdb msg="Deleting mmapped chunk files"
ts=2022-01-26T07:55:39.215Z caller=head.go:671 level=info component=tsdb msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 13233"
ts=2022-01-26T07:55:39.215Z caller=head.go:522 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=21.468515ms
ts=2022-01-26T07:55:39.215Z caller=head.go:528 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-01-26T07:55:42.961Z caller=head.go:564 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2022-01-26T07:55:43.410Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=33 maxSegment=41
ts=2022-01-26T07:55:44.760Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=34 maxSegment=41
ts=2022-01-26T07:55:45.016Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=35 maxSegment=41
ts=2022-01-26T07:55:45.956Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=36 maxSegment=41
ts=2022-01-26T07:55:45.993Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=37 maxSegment=41
ts=2022-01-26T07:55:46.388Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=38 maxSegment=41
ts=2022-01-26T07:55:46.442Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=39 maxSegment=41
ts=2022-01-26T07:55:46.547Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=40 maxSegment=41
ts=2022-01-26T07:55:46.548Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=41 maxSegment=41
ts=2022-01-26T07:55:46.548Z caller=head.go:605 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=3.746433621s wal_replay_duration=3.586436798s total_replay_duration=7.354409599s
ts=2022-01-26T07:55:46.645Z caller=main.go:945 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-01-26T07:55:46.646Z caller=main.go:948 level=info msg="TSDB started"
ts=2022-01-26T07:55:46.646Z caller=main.go:1129 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2022-01-26T07:55:46.647Z caller=main.go:1166 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.635016ms db_storage=2.667µs remote_storage=3.425µs web_handler=1.759µs query_engine=2.5µs scrape=755.445µs scrape_sd=103.868µs notify=2.926µs notify_sd=6.241µs rules=4.204µs
ts=2022-01-26T07:55:46.647Z caller=main.go:897 level=info msg="Server is ready to receive web requests."
oijkn commented 2 years ago

Hi,

This dashboard only graph docker/container stuff not your host. For that you can use other dashboard from Grafana's community.

BR

Akikazuu commented 2 years ago

Hi, thanks for your reply.

After inspecting grafana, i think this graph is the node graph as : image

Docker networks is measured with this : image

The metrics used are not the same, so i think "node" is for the host and "container" is for the docker stuff.

I just found how to resolve that, think its can be usefull for other

Disclamer

I am open to any critical comments, whether in terms of security or good docker practice! Do not hesitate to send me a message via twitter (see on my git profile) or to reply to this thread

docker-compose

You should deploy all containers with : network_mode: host

Prometheus

After in your prometheus config set bellow : 

  - job_name: 'rpi4-1'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090', 'localhost:8080', 'localhost:9100']

You can change "localhost" according to your /etc/hosts file.

Grafana

Finnally in your graphana add in your datasource this line : image You can change "localhost" according to your /etc/hosts file.

Improvement

Hope this thread can help someone who point this issue.

oijkn commented 2 years ago

@Akikazuu give a try to new dashboard version, I have fixed some queries.

oijkn commented 1 year ago

This issue has been inactive for some time. Please let me know if it's still relevant and need attention or I will close it.