henrygd / beszel

Lightweight server monitoring hub with historical data, docker stats, and alerts.
MIT License
1.8k stars 64 forks source link

Fully configurable retention periods #114

Open ghost opened 2 months ago

ghost commented 2 months ago

It would be great if we could have full control over the data retention periods, for example being able to set how often initial data is collected (every 60 seconds for example) and being able to configure how long to retain minutely data and then also how long to retain hourly data etc.

I'd like to store my data for around a year, this would be brilliant if this was a configuration option.

henrygd commented 2 months ago

I don't know if it's a good idea to add this level of configuration. Retaining one minute records for up to a year would increase the size of the database exponentially, possibly leading to performance / scaling issues. Users may not understand the consequences.

The records are not instant-in-time snapshots, but rather averages over the course of the full time period. Even one minute records. So you still get a fairly full picture of utilization without needing to retain hundreds of thousands of individual data points.

If you really need to keep all data for future reference, consider dumping the database to the same file every hour, then snapshotting the file with Kopia or Borg. You can configure retention to keep every one hour backup for a year, and use compression like zstd to keep the backup repository to a reasonable size.

As far as setting how often data is collected, this is a possibility for the future but near the top of the priority list as the current one minute interval works well in my opinion.

Edit: I'm not opposed to adding a one year chart if people want that.

Mark-2445 commented 1 month ago

A year chart would be great!