search

bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform
https://birdhouse-deploy.readthedocs.io/en/latest/
Apache License 2.0
4 stars 6 forks source link

:bulb: [Feature] Add recording rules to Prometheus configuration to store hourly/daily metrics #447

Open huard opened 7 months ago

huard commented 7 months ago

... trying to split the metrics collection issue into small digestible bits.

Description

  1. Select among all existing metrics recorded by Prometheus those of interest to external stakeholders:

    • Number of active users
    • CPU usage
    • Bandwidth usage
    • ...

  2. Create recording rules in our current Prometheus server to store hourly/daily resolution metrics:

    • Total number of active users per day
    • Mean hourly CPU usage
    • Mean hourly bandwidth usage
    • ...

Make sure the frequency is clearly indicated in the metrics names.

  1. Display hourly/daily data in Graphana dashboard

This doesn't solve the data retention issue, but it's a necessary step. A second Prometheus instance can federate the first instance and scrape the metrics that match a hourly/daily regexp pattern.

References

277

Concerned Organizations

mishaschwartz commented 5 months ago

@huard you mentioned that there is an existing metric that records total number of active users per day. Do you know which metric that is (its name, how it is collected)?

huard commented 5 months ago

The only thing I know is that it's part of the vanilla config that came with the docker image. @tlvu would know more about this.

tlvu commented 5 months ago

The only thing I know is that it's part of the vanilla config that came with the docker image. @tlvu would know more about this.

No we do not have that metric (total number of active users per day).

All the metrics that comes with the "vanilla" config are listed here https://github.com/bird-house/birdhouse-deploy/blob/9d9f46c497e2b00a6ad5be9f1e3ec322f85868a3/birdhouse/components/README.rst#grafana-dashboard