Set up Monitoring

[nh758]

“We want to know the memory / cpu usage when a service fails"

Requirements

1. Add promotheus (and grafana?)

Tasks

• [ ] Write Documentation

Reference

Research: https://github.com/digi-serve/ab_runtime/issues/364 https://docs.google.com/document/d/1utvJsxi-smHk1HlYw6kBeLUYqgw56vBTgodip68e-64/edit?usp=drive_link

Promotheus can be set up to monitor docker swarm https://prometheus.io/docs/guides/dockerswarm/ But will it work with podman?

• [x] #387
• [x] #386
• [x] #385

[nh758]

After more research plan is to add these to the stacks we want to monitor:

• cadvisor collects details about running containers
• promethus scrape data from cadvisor and use remote write to send to mimir

Then on a central server run

• Grafana mimir - storage for promethus
• Grafana ui to view the data