Prometheus Monitoring Setup

[jessemillar]

Prometheus/Grafana Server Setup

I use a Raspberry Pi 4 to

1. Install Docker
2. Add user to Docker group
3. mkdir -p ~/prometheus/data && mkdir -p ~/grafana/data
4. Configure Prometheus container:

   docker run -d --restart always --name prometheus -p 9090:9090 -v /home/msa/prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml -v /home/msa/prometheus/data:/opt/bitnami/prometheus/data -u $(id -u):$(id -g) bitnami/prometheus

docker run -d --restart always --name grafana -p 80:3000 -v /home/msa/grafana/data:/var/lib/grafana -u $(id -u):$(id -g) grafana/grafana

docker run -d --restart always --name=speedtest-exporter -p 9876:9876 caarlos0/speedtest-exporter --refresh.interval=5m

docker run -d --restart always --name blackbox-exporter -p 9115/tcp quay.io/prometheus/blackbox-exporter

Node Setup

Linux

1. Follow this guide to install the node_exporter
2. Use the rc.local method on this page to start node_exporter on boot (sudo vi /etc/rc.local and add /home/pi/node_exporter & to the list of commands)

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-armv7.tar.gz
tar -xvzf node_exporter-1.6.0.linux-armv7.tar.gz
mv node_exporter-1.6.0.linux-armv7/node_exporter .
rm -rf node_exporter-*

Windows

1. Install the Windows Prometheus exporter with default options (just run the .msi)
2. Download the Nvidia GPU Prometheus exporter and run it manually once to open the firewall port (if you open your firewall port for "private" networks, make sure the current network is set as "private" in network settings)
   • Follow these guides to set up the Nvidia GPU Exporter on boot without a CMD window appearing. https://winaero.com/run-app-or-script-at-logon-with-task-scheduler-in-windows-10/ and https://www.howtogeek.com/tips/how-to-run-a-scheduled-task-without-a-command-window-appearing/

Add more scraping instances

SSH into the computer running the Prometheus Docker container (the MSA Raspberry Pi):

vi ~/prometheus/prometheus.yml
docker restart prometheus

Making Grafana Dashboards

• When making charts, rate() is different from irate(); rate() is for slow-moving values and irate() is for fast-moving values

[jessemillar]

I need to set up a Docker volume with Grafana so my settings and dashboards aren't inside a container. Here's a guide: https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/#use-bind-mounts

[jessemillar]

I need to figure out how to back up Prometheus and Grafana configs too. Maybe to a private GitHub repo?

[jessemillar]

Here's a link to a method for renaming instances in Prometheus/Grafana to something human-readable: https://stackoverflow.com/a/53053429

[jessemillar]

Collectors Docs

https://github.com/prometheus-community/windows_exporter#collectors https://github.com/utkuozdemir/nvidia_gpu_exporter/blob/master/METRICS.md

[jessemillar]

[jessemillar]

Closed and moved to https://github.com/compycore/msa/issues/1