concourse / hush-house

Concourse k8s-based environment
https://hush-house.pivotal.io
29 stars 23 forks source link

metrics: increase prometheus retention #56

Closed cirocosta closed 4 years ago

cirocosta commented 5 years ago

Hey,

At the moment, the metrics Prometheus server is configured with a retention policy that allows us to store only 15d of data:

Screen Shot 2019-06-17 at 8 48 23 AM

Which, having a default of 15d when 0 (see https://prometheus.io/docs/prometheus/latest/storage/), means that we end up being able to go back a max of 15d.

Even though we didn't have a big need of going way back, it might be useful for some more high-level data like what flight_recorder or an SLI runner could give us.

--

Regarding disk, at the moment we're at a constant 20% utilization:

Screen Shot 2019-06-17 at 8 44 40 AM

My guess is that going up in terms of retention would proportionally make this grow.

cirocosta commented 4 years ago

done https://github.com/concourse/hush-house/issues/77