search

greenstatic / bigbluebutton-exporter

Prometheus exporter for BigBlueButton
https://bigbluebutton-exporter.greenstatic.dev
MIT License
156 stars 162 forks source link

Limit diskusage of database of bigbluebutton-exporter/prometheus/grafana #78

Closed ethpi closed 3 years ago

ethpi commented 3 years ago

I hope this is the right place for my question.

On my BBB-Server I installed the bbb-monitoring "All-In-One Monitoring Stack". Everything works fine. Thanks for your work.

How much diskspace will the database use? Does it grow each day for the same amount? Is there al limit?

The command docker exec -it prometheus du -hs returns 18.0M . Does this mean, that my database uses about 18 MB? I run the monitoring for less than 24h. If the diskusage grows linear, the database uses every month about 600MB. Every year more than 7GB. It's not a really problem, but senseless for me.

How can I limit the diskusage?

greenstatic commented 3 years ago

It's the right place to ask questions :)

The major storage offender is Prometheus since it's a time series database. You can also get a Grafana dashboard for monitoring Prometheus: https://fusakla.github.io/Prometheus2-grafana-dashboard/ - there is a panel that shows exactly how much disk usage is used by the time series db.

Metrics are retained by default for 400 days: https://github.com/greenstatic/bigbluebutton-exporter/blob/2043b331ad2b69d0bc699fe24294b5588ea59d41/extras/all_in_one_monitoring/docker-compose.yaml#L29 You can change this if you wish.

Since you already have all the necessary metrics, you can create an alertmanager (https://prometheus.io/docs/alerting/latest/overview/) rule to alert you when your disk capacity is nearing 80% used storage.

To install alertmanager you can just add another container to the All In One Monitoring Stack docker-compose file since it needs to establish a connection to Prometheus.

In my environment I have the following rules:

The main drawback of having all in one monitoring stack (on the same server as BBB) is that you cannot trigger alerts if the server goes down, or there are any network connectivity issues. So if this is of concern you will need to establish some external service to alert you.

ethpi commented 3 years ago
  • "--storage.tsdb.retention.time=400d"

I will try 31d, thats enough for me. I'm new at docker: to activate the new setting I have to execute 

cd ~/bbb-monitoring && docker-compose down && docker-compose up -d

Correct? Thanks for the alert hints.

greenstatic commented 3 years ago

Correct, that should do it.